2024-09-11T05:11:16.966 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T05:11:17.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T05:11:17.125 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900545' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: daaf78de334647fc0c800f4ac495cc9475a26bae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 6175 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 88/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: daaf78de334647fc0c800f4ac495cc9475a26bae targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4R5Z0t9CyO2ijw5X2bmilZOWw4JS5TyzWNIVmHQavA/VZ5L5JmaD7TGBzKhDgDItX7Ag2MugFVAjbpwj/vOqw= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFNE66bbFJFfY3aglosEWBQvZbm3d6sT9NS8ShDiRZ+Ol7gjme0o88FJVlDbbAzoJQ5eTYq46Sr0aLDb46oVZQ= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T05:11:17.125 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa; will attempt to use it 2024-09-11T05:11:17.126 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa/tasks 2024-09-11T05:11:17.126 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T05:11:17.128 INFO:teuthology.task.internal:Checking packages... 2024-09-11T05:11:17.153 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-11T05:11:17.153 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T05:11:17.153 INFO:teuthology.packaging:ref: None 2024-09-11T05:11:17.153 INFO:teuthology.packaging:tag: None 2024-09-11T05:11:17.153 INFO:teuthology.packaging:branch: squid 2024-09-11T05:11:17.153 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:11:17.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-11T05:11:17.353 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:11:17.355 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T05:11:17.363 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T05:11:17.363 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T05:11:17.376 INFO:teuthology.task.internal:Saving configuration 2024-09-11T05:11:17.393 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T05:11:17.405 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T05:11:17.427 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:05:28.482161', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9pmbFCyYHQO0C5cEwD/wSXN8kHorjJwXcfFCoxmenY'} 2024-09-11T05:11:17.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:05:28.481027', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+PTCpXSuVkbsubeYMb7nKkMbdJEIgkKPOGgeAGDucC'} 2024-09-11T05:11:17.444 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T05:11:17.454 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-11T05:11:17.455 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-11T05:11:17.455 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T05:11:17.517 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f21b6b0f010>, signals=[15]) 2024-09-11T05:11:17.517 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T05:11:17.528 INFO:teuthology.task.internal:Opening connections... 2024-09-11T05:11:17.528 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:11:17.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:11:17.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:11:17.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:11:17.700 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T05:11:17.705 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-09-11T05:11:17.711 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-09-11T05:11:17.712 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-09-11T05:11:17.758 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="22.04" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=jammy 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T05:11:17.759 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=jammy 2024-09-11T05:11:17.760 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-09-11T05:11:17.788 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-09-11T05:11:17.794 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-09-11T05:11:17.794 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="22.04" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=jammy 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T05:11:17.842 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T05:11:17.843 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=jammy 2024-09-11T05:11:17.843 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-09-11T05:11:17.863 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T05:11:17.873 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T05:11:17.880 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T05:11:17.880 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-09-11T05:11:17.882 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-09-11T05:11:17.889 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T05:11:17.896 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T05:11:17.897 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-09-11T05:11:17.928 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-09-11T05:11:17.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T05:11:18.019 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T05:11:18.034 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T05:11:18.035 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T05:11:18.035 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T05:11:18.035 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T05:11:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:11:18.040 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-11T05:11:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:11:18.041 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-09-11T05:11:18.087 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-56-generic 2024-09-11T05:11:18.087 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.15.0-56-generic 2024-09-11T05:11:18.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-09-11T05:11:18.090 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-56-generic 2024-09-11T05:11:18.090 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.15.0-56-generic 2024-09-11T05:11:18.090 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2024-09-11T05:11:18.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-11T05:11:18.228 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-09-11T05:11:18.359 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:11:18.382 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T05:11:18.448 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T05:11:18.476 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T05:11:18.482 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:11:18.482 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T05:11:18.543 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T05:11:18.561 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T05:11:18.667 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T05:11:18.672 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T05:11:18.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T05:11:18.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T05:11:18.697 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T05:11:18.697 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T05:11:18.701 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T05:11:18.701 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T05:11:18.706 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T05:11:18.714 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T05:11:18.716 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T05:11:18.743 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T05:11:18.744 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T05:11:18.744 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T05:11:18.744 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T05:11:18.745 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T05:11:18.745 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T05:11:18.745 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T05:11:18.745 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T05:11:18.761 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T05:11:18.761 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T05:11:18.778 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T05:11:18.778 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T05:11:18.898 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T05:11:18.913 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T05:11:18.923 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T05:11:19.037 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T05:11:19.197 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T05:11:19.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T05:11:19.230 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T05:11:19.368 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T05:11:19.381 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T05:11:19.381 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T05:11:19.383 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T05:11:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T05:11:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T05:11:19.400 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T05:11:19.401 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T05:11:19.401 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T05:11:19.402 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T05:11:19.403 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T05:11:19.468 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T05:11:19.471 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T05:11:19.472 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T05:11:19.476 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T05:11:19.476 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T05:11:19.489 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T05:11:19.490 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T05:11:19.490 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T05:11:19.490 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T05:11:19.497 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T05:11:19.498 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T05:11:19.510 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T05:11:19.520 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T05:11:19.528 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T05:11:19.531 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T05:11:19.533 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T05:11:19.534 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T05:11:19.534 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T05:11:19.540 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T05:11:19.544 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T05:11:19.545 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T05:11:19.548 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T05:11:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T05:11:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T05:11:19.550 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T05:11:19.553 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T05:11:19.554 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T05:11:19.555 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T05:11:19.614 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T05:11:19.628 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T05:11:19.628 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T05:11:19.629 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T05:11:19.630 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T05:11:19.631 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T05:11:19.631 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T05:11:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T05:11:19.635 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T05:11:19.716 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T05:11:19.717 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T05:11:25.009 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 15.8 MB in 2s (7,587 kB/s) 2024-09-11T05:11:26.036 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 15.8 MB in 3s (5,636 kB/s) 2024-09-11T05:11:26.457 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:11:26.475 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:11:26.581 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:11:26.788 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-09-11T05:11:26.789 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-09-11T05:11:26.961 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:11:26.961 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:11:26.961 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:11:26.962 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:11:26.962 INFO:teuthology.orchestra.run.smithi164.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T05:11:26.962 INFO:teuthology.orchestra.run.smithi164.stdout: python2.7-minimal 2024-09-11T05:11:26.962 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:11:26.963 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2024-09-11T05:11:26.963 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:11:26.964 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T05:11:26.964 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T05:11:26.965 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2024-09-11T05:11:26.965 INFO:teuthology.orchestra.run.smithi164.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T05:11:26.965 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2024-09-11T05:11:26.965 INFO:teuthology.orchestra.run.smithi164.stdout: thermald 2024-09-11T05:11:27.023 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2024-09-11T05:11:27.023 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:11:27.023 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T05:11:27.024 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T05:11:27.025 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages will be upgraded: 2024-09-11T05:11:27.025 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T05:11:27.204 INFO:teuthology.orchestra.run.smithi164.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:11:27.204 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 113 MB of archives. 2024-09-11T05:11:27.204 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T05:11:27.204 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T05:11:27.548 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:11:27.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:11:27.678 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:11:27.891 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-11T05:11:27.892 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-11T05:11:28.083 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:11:28.083 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:11:28.083 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:11:28.084 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:11:28.084 INFO:teuthology.orchestra.run.smithi067.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T05:11:28.084 INFO:teuthology.orchestra.run.smithi067.stdout: python2.7-minimal 2024-09-11T05:11:28.084 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:11:28.085 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-09-11T05:11:28.086 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:11:28.086 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T05:11:28.086 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T05:11:28.087 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-09-11T05:11:28.088 INFO:teuthology.orchestra.run.smithi067.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T05:11:28.088 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-09-11T05:11:28.088 INFO:teuthology.orchestra.run.smithi067.stdout: thermald 2024-09-11T05:11:28.139 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-09-11T05:11:28.140 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T05:11:28.140 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T05:11:28.140 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T05:11:28.141 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-09-11T05:11:28.142 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T05:11:28.195 INFO:teuthology.orchestra.run.smithi067.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:11:28.195 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 113 MB of archives. 2024-09-11T05:11:28.195 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T05:11:28.195 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T05:11:28.623 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T05:11:28.671 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T05:11:28.713 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T05:11:29.017 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T05:11:29.485 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T05:11:29.485 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T05:11:29.486 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T05:11:29.609 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T05:11:29.631 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T05:11:30.025 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 113 MB in 1s (76.5 MB/s) 2024-09-11T05:11:30.204 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T05:11:31.205 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T05:11:31.209 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T05:11:31.210 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T05:11:31.555 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T05:11:31.635 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T05:11:32.035 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 113 MB in 5s (24.6 MB/s) 2024-09-11T05:11:32.151 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T05:11:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:32.207 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T05:11:32.234 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:34.172 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T05:11:34.176 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:34.177 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T05:11:34.199 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:34.237 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:34.764 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T05:11:34.785 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:36.322 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T05:11:36.343 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:36.883 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T05:11:36.904 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:36.941 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:41.157 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:41.275 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:41.600 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:41.728 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:42.031 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T05:11:42.054 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T05:11:42.097 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:11:42.965 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:43.375 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:43.496 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:43.790 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T05:11:43.801 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T05:11:43.840 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:11:47.887 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T05:11:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:47.979 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:49.495 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T05:11:49.532 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T05:11:49.570 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:50.925 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:51.250 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:11:51.379 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:51.505 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T05:11:51.634 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:52.165 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T05:11:52.289 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T05:11:52.622 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T05:11:52.752 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:52.878 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T05:11:52.993 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:11:52.993 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T05:11:52.993 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:11:52.993 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T05:11:53.006 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:54.455 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:11:54.455 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T05:11:54.455 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:11:54.455 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T05:11:54.644 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:55.162 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:56.003 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:56.605 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T05:11:56.730 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T05:11:56.857 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:56.914 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T05:11:56.914 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:11:57.388 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T05:11:57.514 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T05:11:57.640 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T05:11:57.706 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T05:11:57.706 INFO:teuthology.orchestra.run.smithi164.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:12:11.012 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T05:12:11.012 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2024-09-11T05:12:11.031 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:12:11.057 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2024-09-11T05:12:11.318 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:12:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:12:11.496 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T05:12:11.497 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub' 2024-09-11T05:12:11.519 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:12:11.551 INFO:teuthology.orchestra.run.smithi164.stdout:Generating grub configuration file ... 2024-09-11T05:12:11.609 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:12:11.614 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:12:11.684 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:12:11.689 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:12:11.843 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:12:11.859 INFO:teuthology.orchestra.run.smithi067.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:12:11.859 INFO:teuthology.orchestra.run.smithi067.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:12:11.859 INFO:teuthology.orchestra.run.smithi067.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:12:11.864 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:12:11.907 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-09-11T05:12:12.219 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:12:12.225 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:12:12.297 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:12:12.302 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:12:12.405 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-09-11T05:12:12.422 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-09-11T05:12:12.422 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.119.119 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-09-11T05:12:12.423 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:12:12.424 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-09-11T05:12:12.424 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T05:12:12.424 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T05:12:12.425 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T05:12:12.425 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-11T05:12:12.425 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T05:12:12.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-09-11T05:12:12.468 INFO:teuthology.orchestra.run.smithi164.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:12:12.468 INFO:teuthology.orchestra.run.smithi164.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:12:12.468 INFO:teuthology.orchestra.run.smithi164.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:12:12.521 INFO:teuthology.orchestra.run.smithi164.stdout:done 2024-09-11T05:12:12.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-11T05:12:12.716 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:12:12.726 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:12:12.743 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:12:12.838 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:12:12.973 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2024-09-11T05:12:13.000 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2024-09-11T05:12:13.000 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 21 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.119.119 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2024-09-11T05:12:13.001 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:12:13.002 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2024-09-11T05:12:13.002 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T05:12:13.002 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T05:12:13.002 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T05:12:13.002 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-11T05:12:13.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T05:12:13.002 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2024-09-11T05:12:13.153 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-09-11T05:12:13.273 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:12:13.395 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:12:13.485 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:12:13.558 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:12:14.160 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:12:14.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:12:14.239 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:12:14.452 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-11T05:12:14.453 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-11T05:12:14.652 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T05:12:14.652 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:12:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:12:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:12:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:12:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:12:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:12:14.653 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:12:14.696 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:12:14.698 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.119.119 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:12:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:12:14.716 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-09-11T05:12:14.716 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-09-11T05:12:14.716 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:12:14.716 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-09-11T05:12:14.716 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-11T05:12:14.737 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:12:14.756 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T05:12:14.761 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.lSEXAmsdlb 2024-09-11T05:12:14.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.lSEXAmsdlb 2024-09-11T05:12:14.822 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:12:14.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.lSEXAmsdlb 2024-09-11T05:12:14.953 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.lSEXAmsdlb is 10KB 2024-09-11T05:12:14.966 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.lSEXAmsdlb 2024-09-11T05:12:14.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.030 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:12:15.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.031 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-09-11T05:12:15.031 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-09-11T05:12:15.097 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T05:12:15.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2024-09-11T05:12:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T05:12:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:12:15.224 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:12:15.224 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:12:15.224 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:12:15.224 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:12:15.225 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:12:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:12:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T05:12:15.269 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 21 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T05:12:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.119.119 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2024-09-11T05:12:15.286 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2024-09-11T05:12:15.286 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-09-11T05:12:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.tp4FM0AqhH 2024-09-11T05:12:15.334 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tp4FM0AqhH 2024-09-11T05:12:15.390 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.tp4FM0AqhH 2024-09-11T05:12:15.502 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.tp4FM0AqhH is 10KB 2024-09-11T05:12:15.515 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.tp4FM0AqhH 2024-09-11T05:12:15.521 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.573 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:12:15.573 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.573 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T05:12:15.641 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T05:12:15.641 DEBUG:teuthology.orchestra.run.smithi164:> sudo update-grub 2024-09-11T05:12:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-09-11T05:12:16.237 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:12:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-09-11T05:12:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:12:16.423 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:12:16.647 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:12:16.652 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:12:16.696 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:12:16.702 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:12:16.835 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:12:16.835 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:12:16.835 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:12:16.846 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-09-11T05:12:16.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-09-11T05:12:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub' 2024-09-11T05:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T05:12:16.870 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-09-11T05:12:17.027 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T05:12:17.033 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T05:12:17.312 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T05:12:17.317 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T05:12:17.361 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T05:12:17.362 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T05:12:17.484 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T05:12:17.484 INFO:teuthology.orchestra.run.smithi164.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T05:12:17.484 INFO:teuthology.orchestra.run.smithi164.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T05:12:17.512 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-09-11T05:12:17.521 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-09-11T05:12:46.889 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-11T05:12:46.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:12:46.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:12:47.524 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-11T05:12:47.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:12:47.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:13:05.341 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-11T05:13:05.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-11T05:13:14.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:13:14.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:13:14.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:13:14.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:13:17.409 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-11T05:13:17.921 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-11T05:13:29.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:13:29.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:13:29.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:13:29.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:14:29.481 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T05:14:29.926 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T05:14:44.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:14:44.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:14:44.754 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-11T05:14:44.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:14:44.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:14:45.282 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-09-11T05:14:45.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:14:45.286 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-119-generic"... 2024-09-11T05:14:45.287 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-11T05:14:45.337 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-119-generic 2024-09-11T05:14:45.337 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T05:14:45.337 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T05:14:45.337 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T05:14:45.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:14:45.810 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-119-generic"... 2024-09-11T05:14:45.810 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-09-11T05:14:45.856 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-119-generic 2024-09-11T05:14:45.856 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T05:14:45.856 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T05:14:45.857 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T05:14:46.338 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-11T05:14:46.339 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T05:14:46.414 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-09-11T05:14:46.435 DEBUG:teuthology.parallel:result is None 2024-09-11T05:14:46.857 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-11T05:14:46.857 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T05:14:46.948 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-09-11T05:14:46.979 DEBUG:teuthology.parallel:result is None 2024-09-11T05:14:46.979 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T05:14:46.987 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T05:14:46.987 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T05:14:46.990 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T05:14:46.997 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T05:14:47.004 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T05:14:47.064 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T05:14:47.064 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T05:14:47.067 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T05:14:47.091 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T05:14:47.097 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T05:14:47.098 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T05:14:47.127 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T05:14:47.145 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:14:47.148 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:14:47.154 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:14:47.156 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T05:14:47.158 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T05:14:47.164 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T05:14:47.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T05:14:47.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T05:14:47.220 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T05:14:47.230 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T05:14:47.230 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T05:14:47.263 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T05:14:47.268 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T05:14:47.314 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T05:14:47.362 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:14:47.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T05:14:47.424 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T05:14:47.430 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T05:14:47.477 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:14:47.477 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T05:14:47.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-09-11T05:14:47.540 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-09-11T05:14:47.616 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T05:14:47.623 INFO:teuthology.task.internal:Starting timer... 2024-09-11T05:14:47.623 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T05:14:47.633 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T05:14:47.642 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-09-11T05:14:47.642 DEBUG:teuthology.task.selinux:Excluding smithi164: OS 'ubuntu' does not support SELinux 2024-09-11T05:14:47.642 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T05:14:47.642 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T05:14:47.642 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T05:14:47.642 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T05:14:47.650 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T05:14:47.662 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T05:14:47.663 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T05:22:19.268 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-09-11T05:22:19.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:22:19.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:22:19.363 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-11T05:22:19.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-11T05:22:19.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:22:19.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:22:19.527 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-09-11T05:22:19.605 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-11T05:22:19.605 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T05:22:19.615 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T05:22:19.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:22:19.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:22:19.619 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:22:19.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:22:19.645 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:22:19.645 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Command line: ntpd -gq 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: ---------------------------------------------------- 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: corporation. Support and training for ntp-4 are 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: available at https://www.nwtime.org/support 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: ---------------------------------------------------- 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: proto: precision = 0.046 usec (-24) 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: basedate set to 2022-02-04 2024-09-11T05:22:19.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:22:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:22:19.647 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:22:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen normally on 4 lo [::1]:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-09-11T05:22:19.649 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:19 ntpd[18443]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:22:19.677 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Command line: ntpd -gq 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: ---------------------------------------------------- 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: corporation. Support and training for ntp-4 are 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: available at https://www.nwtime.org/support 2024-09-11T05:22:19.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: ---------------------------------------------------- 2024-09-11T05:22:19.679 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: proto: precision = 0.044 usec (-24) 2024-09-11T05:22:19.679 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: basedate set to 2022-02-04 2024-09-11T05:22:19.679 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:22:19.679 INFO:teuthology.orchestra.run.smithi164.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:22:19.680 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:22:19.680 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:22:19.681 INFO:teuthology.orchestra.run.smithi164.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen normally on 3 enp3s0f1 172.21.15.164:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen normally on 4 lo [::1]:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:73f3%5]:123 2024-09-11T05:22:19.682 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:19 ntpd[18448]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:20.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:20 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:20.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:20.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:20.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:20.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:20 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:21.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:21.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:21.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:21.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:21.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:21.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:21 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:22 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:22.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:22.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:22.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:22.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:22 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:23.645 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:23 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:23.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:23.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:23 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:24 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:24 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:25 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:25.679 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:25 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:26.646 INFO:teuthology.orchestra.run.smithi067.stdout:11 Sep 05:22:26 ntpd[18443]: ntpd: time slew -0.002111 s 2024-09-11T05:22:26.647 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.002111s 2024-09-11T05:22:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:22:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:11 Sep 05:22:26 ntpd[18443]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:22:26.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:26.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:26.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:22:26.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:22:26.677 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:22:26.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:22:26.678 INFO:teuthology.orchestra.run.smithi164.stdout:11 Sep 05:22:26 ntpd[18448]: ntpd: time slew +0.001203 s 2024-09-11T05:22:26.678 INFO:teuthology.orchestra.run.smithi164.stdout:ntpd: time slew +0.001203s 2024-09-11T05:22:26.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:22:26.678 INFO:teuthology.orchestra.run.smithi164.stderr:11 Sep 05:22:26 ntpd[18448]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:22:26.696 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:22:26.697 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-09-11T05:22:26.697 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.697 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.697 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.697 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.729 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:22:26.729 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2024-09-11T05:22:26.729 INFO:teuthology.orchestra.run.smithi164.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.729 INFO:teuthology.orchestra.run.smithi164.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.730 INFO:teuthology.orchestra.run.smithi164.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.730 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:22:26.730 INFO:teuthology.run_tasks:Running task install... 2024-09-11T05:22:26.738 DEBUG:teuthology.task.install:project ceph 2024-09-11T05:22:26.739 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-11T05:22:26.739 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-11T05:22:26.739 INFO:teuthology.task.install:Using flavor: default 2024-09-11T05:22:26.745 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T05:22:26.745 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-11T05:22:26.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-09-11T05:22:26.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-key list | grep Ceph 2024-09-11T05:22:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:22:26.829 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:22:26.835 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:22:26.836 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:22:26.836 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T05:22:26.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:22:26.856 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:22:26.857 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:22:26.857 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T05:22:26.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:22:26.980 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T05:22:26.980 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:22:26.993 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T05:22:26.993 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:22:27.093 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:22:27.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:22:27.107 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:22:27.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:22:27.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-11T05:22:27.121 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2024-09-11T05:22:27.281 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:22:27.286 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:22:27.286 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:22:27.297 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:22:27.316 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:22:27.337 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:22:27.341 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:22:27.358 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T05:22:27.364 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T05:22:27.380 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:22:27.386 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:22:27.401 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:22:27.401 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T05:22:27.406 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T05:22:27.430 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T05:22:27.435 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T05:22:27.482 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:22:27.564 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:22:27.630 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 29.0 kB in 0s (87.3 kB/s) 2024-09-11T05:22:27.713 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 29.0 kB in 0s (69.8 kB/s) 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:22:28.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:22:28.675 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2024-09-11T05:22:28.761 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:22:28.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:22:28.846 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-11T05:22:28.879 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2024-09-11T05:22:28.880 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2024-09-11T05:22:29.060 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-11T05:22:29.060 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-11T05:22:29.085 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:22:29.085 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:22:29.085 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:22:29.086 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:22:29.086 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:22:29.086 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:22:29.087 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:22:29.087 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2024-09-11T05:22:29.087 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:22:29.088 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:22:29.088 INFO:teuthology.orchestra.run.smithi164.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T05:22:29.088 INFO:teuthology.orchestra.run.smithi164.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T05:22:29.089 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T05:22:29.090 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet zip 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:22:29.091 INFO:teuthology.orchestra.run.smithi164.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2024-09-11T05:22:29.092 INFO:teuthology.orchestra.run.smithi164.stdout: btrfs-tools 2024-09-11T05:22:29.195 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2024-09-11T05:22:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:22:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:22:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:22:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:22:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T05:22:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:22:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:22:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:22:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:22:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:22:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:22:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:22:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T05:22:29.257 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:22:29.257 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 184 MB of archives. 2024-09-11T05:22:29.257 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T05:22:29.257 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:22:29.273 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:22:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:22:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:22:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:22:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:22:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:22:29.274 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:22:29.275 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-09-11T05:22:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:22:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:22:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T05:22:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T05:22:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T05:22:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T05:22:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:22:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet zip 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:22:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:22:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:22:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:22:29.279 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-09-11T05:22:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-09-11T05:22:29.340 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T05:22:29.345 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:22:29.347 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:22:29.364 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:22:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T05:22:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:22:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:22:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T05:22:29.422 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:22:29.427 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:22:29.440 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:22:29.444 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:22:29.444 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:22:29.445 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:22:29.452 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:22:29.452 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 184 MB of archives. 2024-09-11T05:22:29.452 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T05:22:29.453 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:22:29.454 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:22:29.456 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:22:29.458 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:22:29.460 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:22:29.460 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:22:29.463 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:22:29.479 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:22:29.479 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:22:29.480 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:22:29.480 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:22:29.481 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:22:29.482 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:22:29.482 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:22:29.483 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:22:29.495 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:22:29.495 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:22:29.512 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:22:29.514 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:22:29.515 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:22:29.515 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:22:29.516 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:22:29.518 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:22:29.518 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:22:29.518 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:22:29.529 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:22:29.529 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:22:29.541 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T05:22:29.541 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:22:29.542 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T05:22:29.544 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:22:29.546 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:22:29.552 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:22:29.552 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:22:29.552 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:22:29.553 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:22:29.554 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:22:29.557 INFO:teuthology.orchestra.run.smithi164.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:22:29.557 INFO:teuthology.orchestra.run.smithi164.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:22:29.561 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:22:29.563 INFO:teuthology.orchestra.run.smithi164.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:22:29.563 INFO:teuthology.orchestra.run.smithi164.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:22:29.580 INFO:teuthology.orchestra.run.smithi164.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:22:29.580 INFO:teuthology.orchestra.run.smithi164.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:22:29.581 INFO:teuthology.orchestra.run.smithi164.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:22:29.583 INFO:teuthology.orchestra.run.smithi164.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:22:29.588 INFO:teuthology.orchestra.run.smithi164.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T05:22:29.601 INFO:teuthology.orchestra.run.smithi164.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T05:22:29.606 INFO:teuthology.orchestra.run.smithi164.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T05:22:29.606 INFO:teuthology.orchestra.run.smithi164.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T05:22:29.609 INFO:teuthology.orchestra.run.smithi164.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T05:22:29.610 INFO:teuthology.orchestra.run.smithi164.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T05:22:29.616 INFO:teuthology.orchestra.run.smithi164.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T05:22:29.624 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:22:29.629 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:22:29.636 INFO:teuthology.orchestra.run.smithi164.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:22:29.637 INFO:teuthology.orchestra.run.smithi164.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:22:29.639 INFO:teuthology.orchestra.run.smithi164.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:22:29.639 INFO:teuthology.orchestra.run.smithi164.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:22:29.640 INFO:teuthology.orchestra.run.smithi164.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:22:29.642 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:22:29.646 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:22:29.647 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:22:29.648 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:22:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:22:29.658 INFO:teuthology.orchestra.run.smithi164.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:22:29.658 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:22:29.659 INFO:teuthology.orchestra.run.smithi164.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:22:29.659 INFO:teuthology.orchestra.run.smithi164.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:22:29.661 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:22:29.662 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:22:29.664 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:22:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:22:29.674 INFO:teuthology.orchestra.run.smithi164.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:22:29.674 INFO:teuthology.orchestra.run.smithi164.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:22:29.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:22:29.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:22:29.675 INFO:teuthology.orchestra.run.smithi164.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:22:29.676 INFO:teuthology.orchestra.run.smithi164.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:22:29.680 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:22:29.682 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:22:29.683 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:22:29.683 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:22:29.684 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:22:29.685 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:22:29.685 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:22:29.685 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:22:29.689 INFO:teuthology.orchestra.run.smithi164.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:22:29.690 INFO:teuthology.orchestra.run.smithi164.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:22:29.693 INFO:teuthology.orchestra.run.smithi164.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:22:29.698 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:22:29.699 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:22:29.706 INFO:teuthology.orchestra.run.smithi164.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:22:29.707 INFO:teuthology.orchestra.run.smithi164.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:22:29.708 INFO:teuthology.orchestra.run.smithi164.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:22:29.711 INFO:teuthology.orchestra.run.smithi164.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:22:29.713 INFO:teuthology.orchestra.run.smithi164.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:22:29.714 INFO:teuthology.orchestra.run.smithi164.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:22:29.715 INFO:teuthology.orchestra.run.smithi164.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:22:29.716 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:22:29.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:22:29.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:22:29.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:22:29.720 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:22:29.721 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:22:29.721 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:22:29.722 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:22:29.723 INFO:teuthology.orchestra.run.smithi164.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:22:29.724 INFO:teuthology.orchestra.run.smithi164.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:22:29.726 INFO:teuthology.orchestra.run.smithi164.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:22:29.733 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:22:29.734 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:22:29.740 INFO:teuthology.orchestra.run.smithi164.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:22:29.740 INFO:teuthology.orchestra.run.smithi164.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T05:22:29.741 INFO:teuthology.orchestra.run.smithi164.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:22:29.741 INFO:teuthology.orchestra.run.smithi164.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:22:29.742 INFO:teuthology.orchestra.run.smithi164.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T05:22:29.742 INFO:teuthology.orchestra.run.smithi164.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:22:29.743 INFO:teuthology.orchestra.run.smithi164.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:22:29.748 INFO:teuthology.orchestra.run.smithi164.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T05:22:29.749 INFO:teuthology.orchestra.run.smithi164.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:22:29.749 INFO:teuthology.orchestra.run.smithi164.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:22:29.750 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:22:29.756 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:22:29.756 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:22:29.757 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:22:29.757 INFO:teuthology.orchestra.run.smithi164.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:22:29.757 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:22:29.759 INFO:teuthology.orchestra.run.smithi164.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:22:29.759 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:22:29.760 INFO:teuthology.orchestra.run.smithi164.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:22:29.761 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:22:29.762 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:22:29.772 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:22:29.772 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:22:29.788 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:22:29.789 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:22:29.790 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:22:29.793 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:22:29.848 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:22:29.848 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:22:29.850 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:22:29.850 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:22:29.851 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:22:29.869 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:22:29.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:22:29.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:22:29.885 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:22:29.886 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:22:29.886 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:22:29.886 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:22:29.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:22:29.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:22:29.901 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:22:29.902 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:22:29.905 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:22:29.919 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:22:29.920 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:22:29.920 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:22:29.923 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:22:29.926 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:22:29.927 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:22:29.927 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:22:29.936 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:22:29.937 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:22:29.939 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:22:29.953 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:22:29.954 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:22:29.954 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:22:29.955 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:22:29.956 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:22:29.962 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:22:29.962 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:22:29.970 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:22:29.972 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:22:29.973 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:22:30.277 INFO:teuthology.orchestra.run.smithi164.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T05:22:30.326 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T05:22:30.335 INFO:teuthology.orchestra.run.smithi164.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T05:22:30.337 INFO:teuthology.orchestra.run.smithi164.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T05:22:30.338 INFO:teuthology.orchestra.run.smithi164.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T05:22:30.352 INFO:teuthology.orchestra.run.smithi164.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T05:22:30.465 INFO:teuthology.orchestra.run.smithi164.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T05:22:30.760 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T05:22:30.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T05:22:30.878 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T05:22:30.879 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T05:22:30.931 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T05:22:30.932 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T05:22:30.939 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T05:22:30.995 INFO:teuthology.orchestra.run.smithi164.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T05:22:30.995 INFO:teuthology.orchestra.run.smithi164.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T05:22:31.009 INFO:teuthology.orchestra.run.smithi164.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T05:22:31.045 INFO:teuthology.orchestra.run.smithi164.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T05:22:31.054 INFO:teuthology.orchestra.run.smithi164.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T05:22:31.056 INFO:teuthology.orchestra.run.smithi164.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T05:22:31.077 INFO:teuthology.orchestra.run.smithi164.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T05:22:31.254 INFO:teuthology.orchestra.run.smithi164.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T05:22:31.254 INFO:teuthology.orchestra.run.smithi164.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T05:22:31.349 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T05:22:31.352 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T05:22:31.408 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T05:22:32.755 INFO:teuthology.orchestra.run.smithi164.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T05:22:32.757 INFO:teuthology.orchestra.run.smithi164.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T05:22:32.757 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T05:22:32.758 INFO:teuthology.orchestra.run.smithi164.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T05:22:32.836 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T05:22:32.838 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T05:22:32.839 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T05:22:32.880 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T05:22:33.092 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T05:22:33.123 INFO:teuthology.orchestra.run.smithi164.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T05:22:33.125 INFO:teuthology.orchestra.run.smithi164.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T05:22:33.583 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T05:22:33.584 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T05:22:33.594 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T05:22:33.623 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T05:22:33.632 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T05:22:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T05:22:33.655 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T05:22:33.757 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 184 MB in 4s (46.4 MB/s) 2024-09-11T05:22:33.797 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T05:22:33.797 INFO:teuthology.orchestra.run.smithi067.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T05:22:33.923 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:22:34.024 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:22:34.029 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:22:34.121 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:22:34.458 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:22:34.484 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:22:34.525 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:22:34.828 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:22:34.854 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:22:34.856 INFO:teuthology.orchestra.run.smithi067.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T05:22:34.859 INFO:teuthology.orchestra.run.smithi067.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T05:22:34.860 INFO:teuthology.orchestra.run.smithi067.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T05:22:34.894 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:22:35.048 INFO:teuthology.orchestra.run.smithi067.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T05:22:35.049 INFO:teuthology.orchestra.run.smithi067.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T05:22:35.340 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:22:35.366 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:35.415 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:35.694 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 184 MB in 6s (32.3 MB/s) 2024-09-11T05:22:35.793 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:22:35.819 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:35.855 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:22:35.859 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:35.957 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:22:35.963 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:22:36.045 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:22:36.246 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:22:36.272 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:36.312 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:36.382 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:22:36.409 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:22:36.449 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:22:36.649 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:22:36.675 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:22:36.715 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:22:36.785 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:22:36.811 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:22:36.851 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:22:37.035 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librados2. 2024-09-11T05:22:37.061 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:37.101 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:22:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:37.546 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:22:37.572 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:37.612 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:37.677 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:22:37.703 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:37.743 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:38.005 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:22:38.031 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:22:38.048 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:22:38.062 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:22:38.074 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:38.115 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:22:38.419 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:38.432 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:22:38.459 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:38.459 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:22:38.499 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:22:38.778 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:22:38.805 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:38.819 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librados2. 2024-09-11T05:22:38.843 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:38.845 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:38.885 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:39.106 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:22:39.132 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:39.181 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:39.296 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:22:39.323 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:39.379 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:39.525 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:22:39.551 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:39.591 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:39.766 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:22:39.792 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:39.832 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:39.886 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:22:39.912 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:22:39.953 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:22:40.110 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:22:40.125 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:40.168 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:40.298 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:22:40.324 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:22:40.364 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:22:40.487 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:22:40.514 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:40.554 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:40.625 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:22:40.651 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:40.691 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:40.790 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:22:40.816 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:40.848 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:41.019 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:22:41.049 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:22:41.086 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:22:41.126 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:22:41.152 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:41.192 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:41.330 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:22:41.345 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:22:41.389 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:22:41.487 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:22:41.514 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:22:41.554 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:22:41.691 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:22:41.707 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:22:41.750 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:22:41.881 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:22:42.086 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:22:42.113 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:22:42.153 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:22:42.159 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:22:42.186 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:42.226 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:42.448 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:22:42.463 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:22:42.498 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:22:42.570 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:22:42.597 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:22:42.637 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:22:42.725 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package zip. 2024-09-11T05:22:42.740 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:22:42.783 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:22:42.889 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:22:42.905 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:22:42.947 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:22:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package unzip. 2024-09-11T05:22:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:22:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:22:43.259 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:22:43.285 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:22:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:22:43.549 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:22:43.575 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:22:43.615 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:22:43.653 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:22:43.680 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:22:43.720 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:22:43.964 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:22:43.979 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:44.014 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:44.014 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:22:44.042 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:22:44.073 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:22:44.309 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package zip. 2024-09-11T05:22:44.325 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:22:44.367 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:22:44.583 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:22:44.610 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:44.650 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:44.679 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package unzip. 2024-09-11T05:22:44.706 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:22:44.745 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:22:44.969 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:22:44.996 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:22:45.036 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:22:45.132 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:22:45.159 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:22:45.198 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:22:45.340 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:22:45.366 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:45.405 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:45.573 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:22:45.600 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:45.640 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:45.784 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:22:45.811 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:45.850 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:46.192 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:22:46.208 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:46.242 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:46.562 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:22:46.588 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:22:46.628 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:22:46.948 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:22:46.976 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:47.014 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:47.360 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:22:47.386 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:47.426 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:47.522 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:22:47.549 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:47.598 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:48.193 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:22:48.219 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:22:48.259 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:22:48.604 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:22:48.630 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:22:48.670 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:22:48.949 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:22:48.964 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:22:48.975 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:22:48.991 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:49.007 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:22:49.041 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:49.293 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:22:49.319 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:22:49.359 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:22:49.627 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:22:49.643 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:22:49.662 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:22:49.677 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:22:49.688 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:22:49.728 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:22:49.946 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:22:49.973 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:22:50.013 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:22:50.032 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:22:50.058 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:22:50.098 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:22:50.333 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:22:50.342 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:22:50.358 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:22:50.360 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:22:50.392 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:22:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:22:50.688 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:22:50.710 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:22:50.713 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:22:50.737 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:22:50.754 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:22:50.777 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:22:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:22:51.065 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:22:51.082 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:22:51.091 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:22:51.121 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:22:51.131 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:22:51.374 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:22:51.401 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:22:51.440 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:22:51.468 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:22:51.495 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:22:51.526 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:22:51.752 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:22:51.779 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:22:51.780 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:22:51.806 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:22:51.818 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:22:51.846 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:22:52.121 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:22:52.149 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:22:52.149 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:22:52.175 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:22:52.188 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:22:52.215 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:22:52.457 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:22:52.485 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:22:52.516 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:22:52.560 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:22:52.586 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:22:52.626 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:22:52.861 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:22:52.888 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:22:52.927 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:22:52.955 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:22:52.981 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:22:53.021 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:22:53.214 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:22:53.241 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:22:53.266 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:22:53.280 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:22:53.292 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:22:53.332 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:22:53.516 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:22:53.544 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:22:53.583 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:22:53.627 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:22:53.653 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:22:53.693 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:22:53.895 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:22:53.922 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:22:53.961 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:22:54.038 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:22:54.064 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:22:54.107 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:22:54.297 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:22:54.325 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:22:54.364 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:22:54.416 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:22:54.442 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:22:54.482 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:22:54.667 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:22:54.694 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:22:54.733 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:22:54.743 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:22:54.770 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:22:54.810 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:22:54.986 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:22:55.013 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:22:55.052 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:22:55.104 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:22:55.120 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:22:55.154 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:22:55.372 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:22:55.399 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:22:55.424 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:22:55.442 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:22:55.450 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:22:55.490 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:22:55.733 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:22:55.761 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:22:55.800 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:22:55.810 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:22:55.837 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:22:55.876 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:22:56.103 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:22:56.130 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:22:56.169 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:22:56.188 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:22:56.214 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:22:56.254 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:22:56.498 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:22:56.525 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:22:56.564 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:22:56.601 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:22:56.627 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:22:56.659 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:22:56.867 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:22:56.894 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:22:56.933 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:22:56.962 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:22:56.988 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:22:57.028 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:22:57.203 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:22:57.230 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:22:57.261 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:22:57.348 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:22:57.374 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:22:57.414 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:22:57.564 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:22:57.592 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:22:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:22:57.968 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:22:57.977 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:22:57.994 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:22:58.005 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:22:58.034 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:22:58.044 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:22:58.320 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:22:58.335 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:22:58.363 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:22:58.370 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:22:58.391 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:22:58.430 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:22:58.673 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:22:58.689 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:58.732 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:58.741 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:22:58.768 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:22:58.807 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:22:59.086 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:22:59.113 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:22:59.144 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:22:59.145 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:22:59.172 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:22:59.211 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:22:59.447 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:22:59.474 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:22:59.501 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:22:59.513 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:22:59.528 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:22:59.565 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:22:59.800 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:22:59.847 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:22:59.850 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:22:59.866 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:22:59.877 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:22:59.916 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:00.194 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:23:00.221 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:00.260 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:00.315 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:23:00.340 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:00.379 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:00.597 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:23:00.624 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:00.680 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:00.690 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:23:00.718 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:23:00.779 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:23:01.052 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:23:01.079 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:23:01.109 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:23:01.284 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:23:01.311 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:01.350 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:01.479 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:23:01.507 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:01.546 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:01.941 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:23:01.968 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:02.007 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:02.362 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph. 2024-09-11T05:23:02.388 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:02.428 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:02.645 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:23:02.672 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:03.019 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:03.148 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:23:03.175 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:03.531 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:03.868 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:23:03.895 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:03.934 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:04.338 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:23:04.364 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:04.404 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:04.531 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-09-11T05:23:04.558 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:04.597 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:04.774 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:23:04.800 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:23:04.840 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:23:04.917 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:23:04.944 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:04.983 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:05.159 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:23:05.175 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:23:05.218 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:05.329 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:23:05.356 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:05.386 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:05.756 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:23:05.783 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:05.822 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:06.142 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:23:06.169 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:23:06.208 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:23:06.239 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:23:06.267 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:23:06.306 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:06.511 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:23:06.527 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:23:06.569 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:06.617 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:23:06.644 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:23:06.683 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:23:07.028 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:23:07.055 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:07.094 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:07.448 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:23:07.475 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:23:07.515 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:23:07.574 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:23:07.602 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:23:07.641 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:07.792 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:23:07.819 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:23:07.858 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:23:07.960 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:23:07.988 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:23:08.027 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:23:08.380 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:23:08.408 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:08.429 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:23:08.455 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:08.463 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:08.495 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:08.816 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:23:08.844 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:23:08.883 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:23:09.186 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:23:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:23:09.244 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:23:09.547 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:23:09.575 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:09.613 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:09.725 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:23:09.752 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:23:09.791 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:23:10.278 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:23:10.305 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:23:10.345 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:23:10.706 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:23:10.722 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:23:10.765 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:23:10.793 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:23:10.810 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:23:10.852 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:23:11.067 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:23:11.094 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:23:11.134 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:23:11.355 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:23:11.383 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:23:11.422 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:23:11.775 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:23:11.802 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:23:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:23:11.878 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:23:11.905 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:12.021 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:12.144 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:23:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:23:12.211 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:23:12.920 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:23:12.948 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:12.966 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:23:12.993 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:23:13.024 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:23:13.071 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:13.310 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:23:13.326 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:23:13.368 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:23:13.697 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:23:13.724 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:23:13.763 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:23:14.008 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:23:14.032 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:23:14.066 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:23:14.074 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:23:14.102 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:23:14.141 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:23:14.352 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:23:14.380 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:23:14.419 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:23:14.453 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:23:14.480 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:23:14.519 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:23:14.822 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:23:14.849 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:23:14.889 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:23:14.947 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:23:14.975 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:23:15.014 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:23:15.158 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:23:15.174 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:23:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:23:15.300 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:23:15.328 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:23:15.366 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:23:15.511 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:23:15.527 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:23:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:23:15.670 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:23:15.701 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:23:15.737 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:23:16.057 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:23:16.084 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:23:16.123 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:23:16.175 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:23:16.191 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:16.225 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:16.434 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:23:16.462 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:23:16.486 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:23:16.500 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:23:16.514 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:23:16.544 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:23:16.816 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:23:16.831 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:23:16.864 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:23:16.872 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:23:16.899 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:23:16.938 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:23:17.216 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package jq. 2024-09-11T05:23:17.244 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:23:17.282 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:23:17.519 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:23:17.536 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:17.544 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package socat. 2024-09-11T05:23:17.559 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:23:17.569 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:17.593 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:23:17.872 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:23:17.900 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:23:17.913 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:23:17.939 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:23:17.940 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:23:17.980 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:23:18.275 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:23:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:23:18.326 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:23:18.341 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:23:18.353 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:18.392 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:18.594 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-09-11T05:23:18.611 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:23:18.644 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:23:18.855 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-09-11T05:23:18.872 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:23:18.905 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:23:19.217 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:23:19.245 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:23:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:23:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:23:19.666 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:19.703 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:20.671 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:23:20.698 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:20.737 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:21.033 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:23:21.061 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:21.119 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:21.436 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:23:21.464 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:23:21.502 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:23:21.823 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:23:21.851 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:23:21.890 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:23:21.991 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:23:22.019 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T05:23:22.058 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:22.227 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:23:22.254 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:23:22.293 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:23:22.362 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:23:22.378 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:22.420 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:22.715 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:23:22.731 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:23:22.743 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:23:22.760 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:23:22.784 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:23:22.798 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:23:23.043 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:23:23.070 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:23:23.101 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:23:23.102 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:23:23.130 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:23:23.160 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:23:23.531 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:23:23.537 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:23:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:23:23.564 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:23:23.597 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:23:23.603 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:23:23.848 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:23:23.875 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:23:23.941 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:23:24.002 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:23:24.030 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:23:24.068 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:23:24.335 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:23:24.338 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:23:24.355 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:23:24.362 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:23:24.397 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:23:24.401 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:23:24.816 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:23:24.844 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:23:24.883 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:23:24.971 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:23:24.998 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:23:25.037 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:23:25.169 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:23:25.197 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:23:25.235 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:23:25.390 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:23:25.417 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:23:25.447 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:23:25.522 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:23:25.550 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:23:25.589 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:23:25.785 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:23:25.812 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:23:25.851 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:23:25.900 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:23:25.928 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:23:25.966 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:23:26.278 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:23:26.280 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:23:26.305 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:23:26.307 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:23:26.344 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:23:26.346 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:23:26.624 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:23:26.652 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:23:26.673 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:23:26.690 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:23:26.700 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:23:26.739 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:23:27.002 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:23:27.029 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:23:27.076 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:23:27.210 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:23:27.238 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:23:27.276 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:23:27.355 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:23:27.382 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:23:27.421 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:23:27.562 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:23:27.590 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:23:27.629 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:23:27.666 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:23:27.693 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:23:27.732 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:23:27.906 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:23:27.934 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:23:27.973 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:23:28.326 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:23:28.354 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:23:28.393 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:23:28.444 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:23:28.472 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:28.510 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:28.662 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:23:28.678 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:23:28.712 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:23:29.280 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:23:29.296 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:29.355 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:29.391 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:23:29.420 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:29.457 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:29.691 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T05:23:29.719 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:29.757 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:30.212 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:23:30.471 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:23:30.544 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:23:30.572 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:30.597 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:23:30.610 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:30.866 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:23:30.946 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T05:23:30.974 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T05:23:31.012 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:31.193 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:23:31.426 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:23:31.632 INFO:teuthology.orchestra.run.smithi164.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:23:31.676 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:23:31.699 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:31.785 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:23:32.058 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:23:32.215 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user cephadm....done 2024-09-11T05:23:32.371 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:23:32.379 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:23:32.664 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:23:32.778 INFO:teuthology.orchestra.run.smithi067.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:23:32.888 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:32.931 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:23:33.032 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:23:33.284 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:23:33.552 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:23:33.694 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:23:33.754 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-09-11T05:23:33.902 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:23:33.996 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:23:34.187 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:23:34.348 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:23:34.464 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:23:34.590 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:23:34.590 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:23:34.841 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:23:34.858 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:23:35.100 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:23:35.160 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:23:35.209 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:23:35.428 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:23:35.503 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:23:35.695 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:23:36.004 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:36.063 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:23:36.314 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:23:36.331 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:23:36.440 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:23:36.541 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:23:36.590 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:23:36.667 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:23:36.850 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:23:36.935 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:23:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:23:37.369 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:23:37.444 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:23:37.474 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:23:37.563 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:23:37.636 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:37.689 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:23:37.896 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:23:38.022 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:23:38.048 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:38.123 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:23:38.157 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:23:38.249 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:23:38.400 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:23:38.508 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:23:38.660 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:23:38.919 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:23:38.948 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:23:38.969 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:23:39.011 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:23:39.095 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:23:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:23:39.497 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:39.598 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:23:39.730 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:23:39.907 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:23:39.990 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:23:40.099 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:23:40.176 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:23:40.208 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:23:40.485 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:23:40.517 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:23:40.611 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:23:40.894 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:23:41.136 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:23:41.321 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:23:41.513 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:23:41.581 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:23:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:23:41.822 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:23:41.833 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:23:42.134 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:23:42.165 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:23:42.408 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:23:42.527 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:23:42.534 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:42.778 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:23:42.835 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:23:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:23:43.162 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:23:43.481 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:23:43.857 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:23:44.117 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:23:44.234 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:44.367 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:23:44.527 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:23:44.627 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:44.761 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:23:44.862 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:23:45.028 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:23:45.155 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:23:45.280 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:23:45.574 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:23:45.825 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:23:46.059 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:46.185 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:23:46.301 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:23:46.486 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:23:46.552 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:46.686 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:23:46.737 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:23:46.954 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:23:46.997 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:23:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:23:47.215 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:23:47.256 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:23:47.491 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:23:47.641 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:23:47.741 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:23:47.876 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:23:47.950 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:23:48.160 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:23:48.202 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:48.303 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:23:48.327 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:23:48.596 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:23:48.637 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:23:48.897 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:23:48.905 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:23:49.164 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:23:49.181 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:23:49.273 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:23:49.374 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:49.449 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:23:49.471 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:23:49.718 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:23:49.775 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:23:49.869 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:23:50.043 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:23:50.169 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:23:50.270 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:23:50.462 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:23:50.588 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:23:50.855 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:23:50.872 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:23:50.981 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:23:51.224 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:23:51.231 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:51.358 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:23:51.491 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:23:51.601 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:23:51.675 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:23:51.702 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:23:51.791 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:23:52.002 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:23:52.037 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:23:52.163 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:23:52.277 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:23:52.378 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:23:52.654 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:23:52.807 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:23:52.931 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:23:52.933 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:23:53.184 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:23:53.267 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:23:53.310 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:23:53.645 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:23:53.669 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:23:53.937 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:23:53.962 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:23:54.263 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:23:54.272 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:23:54.399 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:54.399 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:23:54.524 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:54.651 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:23:54.666 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:23:54.933 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:23:54.987 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:55.088 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:55.189 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:23:55.262 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:23:55.448 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:55.558 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:55.663 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:23:55.684 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:55.899 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:23:55.899 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:23:55.948 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:23:56.292 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:23:56.369 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:56.418 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:56.528 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:56.544 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:56.654 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:56.670 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:23:56.763 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:23:56.982 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.108 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.234 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:23:57.519 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.645 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.770 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.961 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:57.989 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:23:57.989 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:23:58.087 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.188 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.289 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.424 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.455 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.590 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.716 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:23:58.833 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:23:58.966 INFO:teuthology.orchestra.run.smithi164.stdout:Adding group ceph....done 2024-09-11T05:23:59.351 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user ceph....done 2024-09-11T05:23:59.511 INFO:teuthology.orchestra.run.smithi164.stdout:Setting system user ceph properties....done 2024-09-11T05:23:59.528 INFO:teuthology.orchestra.run.smithi164.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:23:59.663 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:23:59.974 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:24:00.054 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.196 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.331 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.416 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.457 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.544 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.583 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:00.994 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:24:00.994 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:24:01.076 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-09-11T05:24:01.446 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-09-11T05:24:01.485 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:01.614 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-09-11T05:24:01.630 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:24:01.774 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:24:01.837 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:24:02.082 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:24:02.312 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:02.494 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:02.532 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:24:02.532 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:24:02.628 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:02.997 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:03.116 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:24:03.116 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:24:03.197 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T05:24:03.197 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T05:24:03.530 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:03.673 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:03.879 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:24:03.879 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:24:03.890 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:24:04.333 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:04.357 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:04.588 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:24:04.588 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:24:04.608 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:24:04.608 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:24:05.075 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:05.188 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:05.297 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:05.298 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T05:24:05.298 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T05:24:05.448 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:05.659 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:24:05.659 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:24:05.760 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:05.986 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:24:05.986 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:24:06.108 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:06.276 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:06.422 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:06.428 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:06.653 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:06.740 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:24:06.740 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:24:07.180 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:24:07.238 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:07.273 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:24:07.365 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:07.417 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:24:07.525 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:07.743 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:24:07.743 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:24:08.201 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:08.335 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:08.461 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:08.517 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:24:08.620 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T05:24:08.973 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:24:09.074 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:24:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:24:09.159 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:09.226 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:24:10.210 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:24:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:24:10.890 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:10.891 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:24:11.044 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:24:11.063 INFO:teuthology.orchestra.run.smithi067.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:24:11.064 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:24:11.064 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T05:24:11.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T05:24:11.208 DEBUG:teuthology.orchestra.run.smithi164:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:24:11.228 INFO:teuthology.orchestra.run.smithi164.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:24:11.228 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T05:24:11.228 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T05:24:11.230 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T05:24:11.230 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:11.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:24:11.244 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:11.244 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:24:11.282 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T05:24:11.283 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:11.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:24:11.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:24:11.354 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:11.354 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:24:11.367 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:24:11.420 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T05:24:11.421 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:11.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:24:11.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:24:11.493 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:11.635 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:24:11.650 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:24:11.709 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T05:24:11.709 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:11.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:24:11.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:24:11.782 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:11.782 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:24:11.795 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:24:11.848 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T05:24:11.932 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T05:24:11.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-09-11T05:24:11.935 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-09-11T05:24:11.948 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T05:24:11.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:24:11.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:24:12.003 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T05:24:12.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:24:12.048 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:24:12.073 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-11T05:24:12.074 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T05:24:12.074 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900545', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4R5Z0t9CyO2ijw5X2bmilZOWw4JS5TyzWNIVmHQavA/VZ5L5JmaD7TGBzKhDgDItX7Ag2MugFVAjbpwj/vOqw=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFNE66bbFJFfY3aglosEWBQvZbm3d6sT9NS8ShDiRZ+Ol7gjme0o88FJVlDbbAzoJQ5eTYq46Sr0aLDb46oVZQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T05:24:12.074 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T05:24:12.104 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T05:24:12.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:24:12.152 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:24:12.170 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:12.170 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:24:12.211 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:24:12.211 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-11 05:18:46.998291715 +0000 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-11 05:18:46.742294001 +0000 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-11 05:18:46.742294001 +0000 2024-09-11T05:24:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-11T05:24:12.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:24:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-11T05:24:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-11T05:24:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00045046 s, 1.1 MB/s 2024-09-11T05:24:12.315 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:24:12.368 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-09-11T05:24:12.415 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:24:12.415 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:12.415 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-09-11T05:24:12.415 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-11 05:18:47.462287570 +0000 2024-09-11T05:24:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-11 05:18:47.226289678 +0000 2024-09-11T05:24:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-11 05:18:47.226289678 +0000 2024-09-11T05:24:12.416 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-11T05:24:12.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:24:12.469 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-11T05:24:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-11T05:24:12.470 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000412544 s, 1.2 MB/s 2024-09-11T05:24:12.471 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:24:12.521 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-11 05:18:47.958283139 +0000 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-11 05:18:47.694285498 +0000 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-11 05:18:47.694285498 +0000 2024-09-11T05:24:12.567 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-11T05:24:12.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:24:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-11T05:24:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-11T05:24:12.624 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000410887 s, 1.2 MB/s 2024-09-11T05:24:12.625 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:24:12.672 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-11 05:18:48.510278209 +0000 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-11 05:18:48.238280639 +0000 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-11 05:18:48.238280639 +0000 2024-09-11T05:24:12.719 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-11T05:24:12.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:24:12.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-11T05:24:12.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-11T05:24:12.773 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000377797 s, 1.4 MB/s 2024-09-11T05:24:12.775 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:24:12.824 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:12.824 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:24:12.824 INFO:tasks.ceph:Generating config... 2024-09-11T05:24:12.829 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T05:24:12.829 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:12.830 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:12.831 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:12.831 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:12.831 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:12.831 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:12.831 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:12.831 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T05:24:12.832 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T05:24:12.832 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T05:24:12.832 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T05:24:12.832 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T05:24:12.832 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T05:24:12.832 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T05:24:12.833 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T05:24:12.833 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T05:24:12.834 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T05:24:12.834 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-11T05:24:12.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-11T05:24:12.900 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster1.keyring 2024-09-11T05:24:12.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-11T05:24:12.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T05:24:13.041 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.67')] 2024-09-11T05:24:13.042 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T05:24:13.043 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:13.043 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:24:13.091 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:13.171 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:13.171 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid aeb205b2-dc94-4e51-98a0-6978e8bdb941 2024-09-11T05:24:13.171 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-09-11T05:24:13.171 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:fsid aeb205b2-dc94-4e51-98a0-6978e8bdb941 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-09-11T05:24:13.172361+0000 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-09-11T05:24:13.172361+0000 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-09-11T05:24:13.172 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-11T05:24:13.173 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:24:13.218 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID aeb205b2-dc94-4e51-98a0-6978e8bdb941... 2024-09-11T05:24:13.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T05:24:13.264 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T05:24:13.293 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-09-11T05:24:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-09-11T05:24:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:24:13.293 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:24:13.294 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:24:13.295 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-09-11T05:24:13.296 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = 172.21.15.67 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = aeb205b2-dc94-4e51-98a0-6978e8bdb941 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:24:13.297 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.298 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression algorithm = lz4 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression mode = aggressive 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-09-11T05:24:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-09-11T05:24:13.300 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-09-11T05:24:13.301 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-09-11T05:24:13.302 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd = 20 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd_mirror = 15 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: log to stderr = False 2024-09-11T05:24:13.303 INFO:teuthology.orchestra.run.smithi067.stdout: rbd default features = 125 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: rbd mirror image policy type = none 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-09-11T05:24:13.304 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-09-11T05:24:13.305 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-09-11T05:24:13.306 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:24:13.307 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-09-11T05:24:13.308 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.67 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 5000 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = aeb205b2-dc94-4e51-98a0-6978e8bdb941 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:24:13.309 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.310 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression algorithm = lz4 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression mode = aggressive 2024-09-11T05:24:13.311 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-09-11T05:24:13.312 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-11T05:24:13.313 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:24:13.314 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd = 20 2024-09-11T05:24:13.315 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd_mirror = 15 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: log to stderr = False 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: rbd default features = 125 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: rbd mirror image policy type = none 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.316 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2024-09-11T05:24:13.317 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:13.317 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:13.318 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-11T05:24:13.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-11T05:24:13.409 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T05:24:13.410 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:13.410 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-11T05:24:13.415 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:13.416 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-11T05:24:13.463 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:24:13.463 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:13.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T05:24:13.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T05:24:13.527 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:13.528 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:13.575 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:24:13.575 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:13.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T05:24:13.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T05:24:13.597 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:13.598 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:13.649 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T05:24:13.649 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T05:24:13.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T05:24:13.694 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T05:24:13.697 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T05:24:13.698 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T05:24:13.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-11T05:24:13.732 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-11T05:24:13.742 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T05:24:13.778 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T05:24:13.797 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T05:24:13.874 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T05:24:13.884 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T05:24:13.956 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T05:24:13.966 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T05:24:14.041 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T05:24:14.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T05:24:14.127 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T05:24:14.140 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T05:24:14.211 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T05:24:14.222 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T05:24:14.296 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T05:24:14.307 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T05:24:14.381 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T05:24:14.390 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T05:24:14.391 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:24:14.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-11T05:24:14.404 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:14.405 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-11T05:24:14.405 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:24:14.405 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:14.468 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:14.469 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:14.929 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-11T05:24:14.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-11T05:24:14.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-11T05:24:15.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-11T05:24:15.039 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:15.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-11T05:24:15.092 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:15.092 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-11T05:24:15.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:24:15.092 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:24:15.155 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:15.156 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:15.621 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-11T05:24:15.622 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-11T05:24:15.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-11T05:24:15.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-11T05:24:15.773 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:15.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-11T05:24:15.831 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:15.831 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-11T05:24:15.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:24:15.831 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:24:15.894 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:15.896 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:16.369 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-09-11T05:24:16.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-11T05:24:16.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-11T05:24:16.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-11T05:24:16.525 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:16.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:16.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:16.642+0000 7f4472f807c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-11T05:24:16.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:16.642+0000 7f4472f807c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-11T05:24:16.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:16.642+0000 7f4472f807c0 -1 bdev(0x5580129ab800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-11T05:24:16.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:16.642+0000 7f4472f807c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-11T05:24:19.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-11T05:24:19.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:19.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:19.882+0000 7f137b4eb7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-11T05:24:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:19.886+0000 7f137b4eb7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-11T05:24:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:19.886+0000 7f137b4eb7c0 -1 bdev(0x5575c9f41800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-11T05:24:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:19.886+0000 7f137b4eb7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-11T05:24:23.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-11T05:24:23.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:23.174+0000 7f4663b1f7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-11T05:24:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:23.174+0000 7f4663b1f7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-11T05:24:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:23.174+0000 7f4663b1f7c0 -1 bdev(0x5646b9e01800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-11T05:24:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-11T05:24:23.174+0000 7f4663b1f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-11T05:24:26.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-11T05:24:26.403 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T05:24:26.403 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:26.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-11T05:24:26.457 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:26.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-11T05:24:26.514 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:26.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-11T05:24:26.570 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:26.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-11T05:24:26.626 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:26.626 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-11T05:24:26.675 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.675 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-11T05:24:26.683 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.683 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-11T05:24:26.729 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.730 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-11T05:24:26.777 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.778 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-11T05:24:26.826 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.826 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-11T05:24:26.873 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.874 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-11T05:24:26.921 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.922 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-11T05:24:26.969 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:26.970 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-11T05:24:27.017 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T05:24:27.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9KeFmNVx1KRAActyzbVIRpUumWJWJULaHhA== 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCAKeFmU7NtJhAAL89JH2dUhZ+oLawS6W1K5A== 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-09-11T05:24:27.030 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCDKeFmhKzaNBAAWe01Znuktylju3ZVPhAh0A== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCHKeFmqd13ChAAom8IKeWz03EHDjGgf3Katg== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9KeFmGF2tKxAA4WJSsMlrICYzcdQU30Mu3Q== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9KeFmuVp5LhAAZEy88J9f18FBYw1y+/9WlA== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9KeFmt8YyNBAA+FYVbM07tFO8nyXn+0Ya0Q== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9KeFmsGMLORAAEOcrnekcUxgik6YRN6R1fA== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+KeFmqoCNAhAAy2RjZHiu7AlUvx3t0u8Ouw== 2024-09-11T05:24:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+KeFmE7SoBxAAwKJO4TTkUCritMyYKBg3aw== 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.4] 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+KeFm+LOpDBAA3z5VVS8KVle5ANCgjhIciQ== 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.5] 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+KeFm4R61ERAA6dAoybhLCHL2RKKGMfbdUQ== 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.6] 2024-09-11T05:24:27.032 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB+KeFmdnvGFhAAJjtOYzLmeMEhqvH4biO6kw== 2024-09-11T05:24:27.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:24:27.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:24:27.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:24:27.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:24:27.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:24:27.918 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T05:24:27.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-11T05:24:27.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-11T05:24:28.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-11T05:24:28.082 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T05:24:28.130 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-11T05:24:28.131 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-11T05:24:28.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-11T05:24:28.176 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-11T05:24:28.176 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-11T05:24:28.176 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-11T05:24:28.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-11T05:24:28.179 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-11T05:24:28.179 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T05:24:28.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-11T05:24:28.425 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-09-11T05:24:28.444 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T05:24:28.444 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-11T05:24:28.444 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:28.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-11T05:24:28.455 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:28.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-11T05:24:28.507 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:28.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-11T05:24:28.550 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:28.550+0000 7fc764dbc180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T05:24:28.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new 2c33b7ad-c171-474c-8a11-23bdde4e7282 0 2024-09-11T05:24:28.803 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-09-11T05:24:28.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new f39a9aee-6278-415d-bf58-d41b43525a6d 1 2024-09-11T05:24:29.021 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-09-11T05:24:29.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd new 686e2789-7c26-4e0e-bec8-516001c9645e 2 2024-09-11T05:24:29.062 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:29.062+0000 7fc764dbc180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T05:24:29.238 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-09-11T05:24:29.256 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-11T05:24:29.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-11T05:24:29.260 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:29.258+0000 7fc764dbc180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T05:24:29.261 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-11T05:24:29.261 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-11T05:24:29.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-11T05:24:29.263 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-11T05:24:29.263 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-11T05:24:29.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-11T05:24:29.265 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-11T05:24:29.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:29.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:29.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:29.239161+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:29.462 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-11T05:24:29.463 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T05:24:29.765 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:29.878 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:29.878+0000 7fc764dbc180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T05:24:29.937 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:29.934+0000 7fc764dbc180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T05:24:29.948 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-11T05:24:29.948 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":5,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:29.239161+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:29.963 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T05:24:29.994 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:29.994+0000 7fc764dbc180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T05:24:30.075 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.074+0000 7fc764dbc180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T05:24:30.130 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.130+0000 7fc764dbc180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T05:24:30.211 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.210+0000 7fc764dbc180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T05:24:30.267 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.266+0000 7fc764dbc180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T05:24:30.411 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:24:30.410+0000 7f67d56e37c0 -1 Falling back to public interface 2024-09-11T05:24:30.414 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-09-11T05:24:30.414+0000 7f0f138537c0 -1 Falling back to public interface 2024-09-11T05:24:30.423 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-09-11T05:24:30.422+0000 7f5c546bd7c0 -1 Falling back to public interface 2024-09-11T05:24:30.426 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.426+0000 7fc764dbc180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T05:24:30.492 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:30.490+0000 7fc764dbc180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T05:24:31.148 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T05:24:31.148 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T05:24:31.149 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-09-11T05:24:31.221 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:31.218+0000 7fc764dbc180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T05:24:31.411 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:31.410+0000 7fc764dbc180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:24:31.566 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:31.566+0000 7fc764dbc180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T05:24:31.633 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:31.630+0000 7fc764dbc180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T05:24:31.775 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:31.774+0000 7fc764dbc180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:24:32.401 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:32.398+0000 7fc764dbc180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T05:24:32.481 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:32.478+0000 7fc764dbc180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T05:24:32.674 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:32.674+0000 7fc764dbc180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T05:24:32.751 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:32.750+0000 7fc764dbc180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T05:24:33.059 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:33.058+0000 7fc764dbc180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T05:24:33.124 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:33.122+0000 7fc764dbc180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T05:24:34.155 INFO:tasks.ceph.cluster1.mgr.x.smithi067.stderr:2024-09-11T05:24:34.154+0000 7fc70b8f0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T05:24:34.629 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:24:34.626+0000 7f67d56e37c0 -1 osd.0 0 log_to_monitors true 2024-09-11T05:24:34.700 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-09-11T05:24:34.698+0000 7f0f138537c0 -1 osd.1 0 log_to_monitors true 2024-09-11T05:24:35.244 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-09-11T05:24:35.242+0000 7f5c546bd7c0 -1 osd.2 0 log_to_monitors true 2024-09-11T05:24:36.266 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:36.678 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-11T05:24:36.678 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":7,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:36.203886+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:36.700 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T05:24:37.191 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-09-11T05:24:37.190+0000 7f0f0f7fc640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T05:24:37.219 INFO:tasks.ceph.cluster1.osd.1.smithi067.stderr:2024-09-11T05:24:37.218+0000 7f0f07631640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:24:37.226 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:24:37.226+0000 7f67d168c640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T05:24:37.250 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:24:37.250+0000 7f67c94c1640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:24:38.063 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-09-11T05:24:38.062+0000 7f5c50666640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T05:24:38.083 INFO:tasks.ceph.cluster1.osd.2.smithi067.stderr:2024-09-11T05:24:38.082+0000 7f5c4849b640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:24:43.005 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:43.312 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-11T05:24:43.312 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:41.335575+0000","last_up_change":"2024-09-11T05:24:38.228180+0000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:24:39.240696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6801","nonce":4088995954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6803","nonce":4088995954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6807","nonce":4088995954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6805","nonce":4088995954}]},"public_addr":"172.21.15.67:6801/4088995954","cluster_addr":"172.21.15.67:6803/4088995954","heartbeat_back_addr":"172.21.15.67:6807/4088995954","heartbeat_front_addr":"172.21.15.67:6805/4088995954","state":["exists","up"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6809","nonce":2056339409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6811","nonce":2056339409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6815","nonce":2056339409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6813","nonce":2056339409}]},"public_addr":"172.21.15.67:6809/2056339409","cluster_addr":"172.21.15.67:6811/2056339409","heartbeat_back_addr":"172.21.15.67:6815/2056339409","heartbeat_front_addr":"172.21.15.67:6813/2056339409","state":["exists","up"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6817","nonce":4074589272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6819","nonce":4074589272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6823","nonce":4074589272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6821","nonce":4074589272}]},"public_addr":"172.21.15.67:6817/4074589272","cluster_addr":"172.21.15.67:6819/4074589272","heartbeat_back_addr":"172.21.15.67:6823/4074589272","heartbeat_front_addr":"172.21.15.67:6821/4074589272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.591349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.668801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:36.227070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:43.329 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T05:24:43.330 INFO:tasks.ceph:Creating RBD pool 2024-09-11T05:24:43.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-11T05:24:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2024-09-11T05:24:44.375 DEBUG:teuthology.orchestra.run.smithi067:> rbd --cluster cluster1 pool init rbd 2024-09-11T05:24:47.408 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-11T05:24:47.409 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-11T05:24:47.409 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-11T05:24:47.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-11T05:24:47.410 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T05:24:47.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:47.780 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2241041971},{"type":"v1","addr":"172.21.15.67:6825","nonce":2241041971}]},"active_addr":"172.21.15.67:6825/2241041971","active_change":"2024-09-11T05:24:33.144445+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":70458507}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":394073410}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":284998186}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3119329357}]}]} 2024-09-11T05:24:47.783 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-11T05:24:47.783 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-11T05:24:47.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:48.073 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:47.363780+0000","last_up_change":"2024-09-11T05:24:38.228180+0000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:24:39.240696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:24:43.666844+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6801","nonce":4088995954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6803","nonce":4088995954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6807","nonce":4088995954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6805","nonce":4088995954}]},"public_addr":"172.21.15.67:6801/4088995954","cluster_addr":"172.21.15.67:6803/4088995954","heartbeat_back_addr":"172.21.15.67:6807/4088995954","heartbeat_front_addr":"172.21.15.67:6805/4088995954","state":["exists","up"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6809","nonce":2056339409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6811","nonce":2056339409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6815","nonce":2056339409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6813","nonce":2056339409}]},"public_addr":"172.21.15.67:6809/2056339409","cluster_addr":"172.21.15.67:6811/2056339409","heartbeat_back_addr":"172.21.15.67:6815/2056339409","heartbeat_front_addr":"172.21.15.67:6813/2056339409","state":["exists","up"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6817","nonce":4074589272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6819","nonce":4074589272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6823","nonce":4074589272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6821","nonce":4074589272}]},"public_addr":"172.21.15.67:6817/4074589272","cluster_addr":"172.21.15.67:6819/4074589272","heartbeat_back_addr":"172.21.15.67:6823/4074589272","heartbeat_front_addr":"172.21.15.67:6821/4074589272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.591349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.668801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:36.227070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:48.091 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-11T05:24:48.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T05:24:48.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:48.378 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T05:24:47.363780+0000","last_up_change":"2024-09-11T05:24:38.228180+0000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:24:39.240696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:24:43.666844+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6801","nonce":4088995954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6803","nonce":4088995954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6807","nonce":4088995954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6805","nonce":4088995954}]},"public_addr":"172.21.15.67:6801/4088995954","cluster_addr":"172.21.15.67:6803/4088995954","heartbeat_back_addr":"172.21.15.67:6807/4088995954","heartbeat_front_addr":"172.21.15.67:6805/4088995954","state":["exists","up"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6809","nonce":2056339409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6811","nonce":2056339409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6815","nonce":2056339409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6813","nonce":2056339409}]},"public_addr":"172.21.15.67:6809/2056339409","cluster_addr":"172.21.15.67:6811/2056339409","heartbeat_back_addr":"172.21.15.67:6815/2056339409","heartbeat_front_addr":"172.21.15.67:6813/2056339409","state":["exists","up"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6817","nonce":4074589272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6819","nonce":4074589272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6823","nonce":4074589272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6821","nonce":4074589272}]},"public_addr":"172.21.15.67:6817/4074589272","cluster_addr":"172.21.15.67:6819/4074589272","heartbeat_back_addr":"172.21.15.67:6823/4074589272","heartbeat_front_addr":"172.21.15.67:6821/4074589272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.591349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.668801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:36.227070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:24:48.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-11T05:24:48.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-11T05:24:48.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-11T05:24:48.547 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:48.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T05:24:48.548 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:48.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T05:24:48.549 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:48.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T05:24:48.870 INFO:teuthology.orchestra.run.smithi067.stdout:38654705668 2024-09-11T05:24:48.877 INFO:teuthology.orchestra.run.smithi067.stdout:38654705668 2024-09-11T05:24:48.889 INFO:teuthology.orchestra.run.smithi067.stdout:38654705668 2024-09-11T05:24:48.890 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.0 2024-09-11T05:24:48.898 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.2 2024-09-11T05:24:48.909 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.1 2024-09-11T05:24:49.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T05:24:49.899 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T05:24:49.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T05:24:50.206 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:50.223 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.0 2024-09-11T05:24:50.223 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:50.233 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:50.250 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.2 2024-09-11T05:24:50.251 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:50.269 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-09-11T05:24:50.286 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.1 2024-09-11T05:24:50.286 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:50.286 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T05:24:50.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T05:24:50.605 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T05:24:50.621 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-11T05:24:49.166253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85788,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283029732,"statfs":{"total":289910292480,"available":289822445568,"internally_reserved":0,"allocated":540672,"data_stored":1302373,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391805+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T05:24:47.391805+0000","last_peered":"2024-09-11T05:24:47.391805+0000","last_clean":"2024-09-11T05:24:47.391805+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T05:24:47.391805+0000","last_undegraded":"2024-09-11T05:24:47.391805+0000","last_fullsized":"2024-09-11T05:24:47.391805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.392518+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T05:24:47.392518+0000","last_peered":"2024-09-11T05:24:47.392518+0000","last_clean":"2024-09-11T05:24:47.392518+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T05:24:47.392518+0000","last_undegraded":"2024-09-11T05:24:47.392518+0000","last_fullsized":"2024-09-11T05:24:47.392518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391169+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T05:24:47.391169+0000","last_peered":"2024-09-11T05:24:47.391169+0000","last_clean":"2024-09-11T05:24:47.391169+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T05:24:47.391169+0000","last_undegraded":"2024-09-11T05:24:47.391169+0000","last_fullsized":"2024-09-11T05:24:47.391169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391956+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T05:24:47.391956+0000","last_peered":"2024-09-11T05:24:47.391956+0000","last_clean":"2024-09-11T05:24:47.391956+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T05:24:47.391956+0000","last_undegraded":"2024-09-11T05:24:47.391956+0000","last_fullsized":"2024-09-11T05:24:47.391956+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.386928+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T05:24:47.386928+0000","last_peered":"2024-09-11T05:24:47.386928+0000","last_clean":"2024-09-11T05:24:47.386928+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T05:24:47.386928+0000","last_undegraded":"2024-09-11T05:24:47.386928+0000","last_fullsized":"2024-09-11T05:24:47.386928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.377777+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T05:24:47.377777+0000","last_peered":"2024-09-11T05:24:47.377777+0000","last_clean":"2024-09-11T05:24:47.377777+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T05:24:47.377777+0000","last_undegraded":"2024-09-11T05:24:47.377777+0000","last_fullsized":"2024-09-11T05:24:47.377777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.377540+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T05:24:47.377540+0000","last_peered":"2024-09-11T05:24:47.377540+0000","last_clean":"2024-09-11T05:24:47.377540+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T05:24:47.377540+0000","last_undegraded":"2024-09-11T05:24:47.377540+0000","last_fullsized":"2024-09-11T05:24:47.377540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.390942+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T05:24:47.390942+0000","last_peered":"2024-09-11T05:24:47.390942+0000","last_clean":"2024-09-11T05:24:47.390942+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T05:24:47.390942+0000","last_undegraded":"2024-09-11T05:24:47.390942+0000","last_fullsized":"2024-09-11T05:24:47.390942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.389851+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T05:24:47.389851+0000","last_peered":"2024-09-11T05:24:47.389851+0000","last_clean":"2024-09-11T05:24:47.389851+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T05:24:47.389851+0000","last_undegraded":"2024-09-11T05:24:47.389851+0000","last_fullsized":"2024-09-11T05:24:47.389851+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":630920,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":200704,"data_stored":630920,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:24:50.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T05:24:50.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:50.906 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T05:24:50.922 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-11T05:24:49.166253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85788,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283029732,"statfs":{"total":289910292480,"available":289822445568,"internally_reserved":0,"allocated":540672,"data_stored":1302373,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.817726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391805+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T05:24:47.391805+0000","last_peered":"2024-09-11T05:24:47.391805+0000","last_clean":"2024-09-11T05:24:47.391805+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T05:24:47.391805+0000","last_undegraded":"2024-09-11T05:24:47.391805+0000","last_fullsized":"2024-09-11T05:24:47.391805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.392518+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T05:24:47.392518+0000","last_peered":"2024-09-11T05:24:47.392518+0000","last_clean":"2024-09-11T05:24:47.392518+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T05:24:47.392518+0000","last_undegraded":"2024-09-11T05:24:47.392518+0000","last_fullsized":"2024-09-11T05:24:47.392518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391169+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T05:24:47.391169+0000","last_peered":"2024-09-11T05:24:47.391169+0000","last_clean":"2024-09-11T05:24:47.391169+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T05:24:47.391169+0000","last_undegraded":"2024-09-11T05:24:47.391169+0000","last_fullsized":"2024-09-11T05:24:47.391169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.391956+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T05:24:47.391956+0000","last_peered":"2024-09-11T05:24:47.391956+0000","last_clean":"2024-09-11T05:24:47.391956+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T05:24:47.391956+0000","last_undegraded":"2024-09-11T05:24:47.391956+0000","last_fullsized":"2024-09-11T05:24:47.391956+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.386928+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T05:24:47.386928+0000","last_peered":"2024-09-11T05:24:47.386928+0000","last_clean":"2024-09-11T05:24:47.386928+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T05:24:47.386928+0000","last_undegraded":"2024-09-11T05:24:47.386928+0000","last_fullsized":"2024-09-11T05:24:47.386928+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.377777+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T05:24:47.377777+0000","last_peered":"2024-09-11T05:24:47.377777+0000","last_clean":"2024-09-11T05:24:47.377777+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T05:24:47.377777+0000","last_undegraded":"2024-09-11T05:24:47.377777+0000","last_fullsized":"2024-09-11T05:24:47.377777+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.377540+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T05:24:47.377540+0000","last_peered":"2024-09-11T05:24:47.377540+0000","last_clean":"2024-09-11T05:24:47.377540+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T05:24:47.377540+0000","last_undegraded":"2024-09-11T05:24:47.377540+0000","last_fullsized":"2024-09-11T05:24:47.377540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.390942+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T05:24:47.390942+0000","last_peered":"2024-09-11T05:24:47.390942+0000","last_clean":"2024-09-11T05:24:47.390942+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T05:24:47.390942+0000","last_undegraded":"2024-09-11T05:24:47.390942+0000","last_fullsized":"2024-09-11T05:24:47.390942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:24:47.389851+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T05:24:47.389851+0000","last_peered":"2024-09-11T05:24:47.389851+0000","last_clean":"2024-09-11T05:24:47.389851+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T05:24:47.389851+0000","last_undegraded":"2024-09-11T05:24:47.389851+0000","last_fullsized":"2024-09-11T05:24:47.389851+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":200704,"data_stored":630920,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":200704,"data_stored":630920,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:24:50.923 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T05:24:50.923 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-11T05:24:50.923 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-11T05:24:50.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-11T05:24:51.240 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:51.240 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T05:24:51.258 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-11T05:24:51.258 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T05:24:51.268 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-11T05:24:51.268 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T05:24:51.268 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900545', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4R5Z0t9CyO2ijw5X2bmilZOWw4JS5TyzWNIVmHQavA/VZ5L5JmaD7TGBzKhDgDItX7Ag2MugFVAjbpwj/vOqw=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOFNE66bbFJFfY3aglosEWBQvZbm3d6sT9NS8ShDiRZ+Ol7gjme0o88FJVlDbbAzoJQ5eTYq46Sr0aLDb46oVZQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T05:24:51.269 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T05:24:51.271 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T05:24:51.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:24:51.319 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:24:51.334 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:51.334 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:24:51.381 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:24:51.382 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-11 05:18:45.778396679 +0000 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-11 05:18:45.534354938 +0000 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-11 05:18:45.534354938 +0000 2024-09-11T05:24:51.430 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-09-11T05:24:51.430 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:24:51.484 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-11T05:24:51.484 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-11T05:24:51.484 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000448363 s, 1.1 MB/s 2024-09-11T05:24:51.486 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:24:51.535 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-09-11T05:24:51.581 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:24:51.581 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-11 05:18:46.274479919 +0000 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-11 05:18:46.014437002 +0000 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-11 05:18:46.014437002 +0000 2024-09-11T05:24:51.582 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-09-11T05:24:51.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:24:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-11T05:24:51.636 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-11T05:24:51.636 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000386127 s, 1.3 MB/s 2024-09-11T05:24:51.637 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:24:51.686 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-11 05:18:46.510518875 +0000 2024-09-11T05:24:51.733 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-11 05:18:46.506518215 +0000 2024-09-11T05:24:51.734 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-11 05:18:46.506518215 +0000 2024-09-11T05:24:51.734 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-09-11T05:24:51.734 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:24:51.787 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-11T05:24:51.787 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-11T05:24:51.787 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000414517 s, 1.2 MB/s 2024-09-11T05:24:51.789 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:24:51.838 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-09-11T05:24:51.885 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:24:51.885 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:24:51.885 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-11T05:24:51.886 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:24:51.886 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-11 05:18:47.262641726 +0000 2024-09-11T05:24:51.886 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-11 05:18:46.990598107 +0000 2024-09-11T05:24:51.886 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-11 05:18:46.990598107 +0000 2024-09-11T05:24:51.886 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2024-09-11T05:24:51.886 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:24:51.939 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-11T05:24:51.939 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-11T05:24:51.939 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000306673 s, 1.7 MB/s 2024-09-11T05:24:51.940 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:24:51.987 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:51.987 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:24:51.987 INFO:tasks.ceph:Generating config... 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:51.988 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:51.989 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:51.989 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T05:24:51.989 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T05:24:51.989 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T05:24:51.989 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T05:24:51.989 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T05:24:51.989 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T05:24:51.989 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T05:24:51.989 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T05:24:51.990 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T05:24:51.991 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T05:24:51.991 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-11T05:24:51.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.059 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.062 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.138 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.192 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.164')] 2024-09-11T05:24:52.193 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.164', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T05:24:52.194 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:52.194 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:24:52.241 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.164 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:52.320 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:52.320 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: generated fsid 53ed04a9-aee2-4881-a1db-0ad67faa19f3 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:setting min_mon_release = quincy 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:epoch 0 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:fsid 53ed04a9-aee2-4881-a1db-0ad67faa19f3 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:last_changed 2024-09-11T05:24:52.321431+0000 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:created 2024-09-11T05:24:52.321431+0000 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:min_mon_release 17 (quincy) 2024-09-11T05:24:52.321 INFO:teuthology.orchestra.run.smithi164.stdout:election_strategy: 1 2024-09-11T05:24:52.322 INFO:teuthology.orchestra.run.smithi164.stdout:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-11T05:24:52.322 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-11T05:24:52.323 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:24:52.370 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 53ed04a9-aee2-4881-a1db-0ad67faa19f3... 2024-09-11T05:24:52.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T05:24:52.373 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T05:24:52.415 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-09-11T05:24:52.416 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-09-11T05:24:52.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.418 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = 172.21.15.164 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-09-11T05:24:52.419 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 53ed04a9-aee2-4881-a1db-0ad67faa19f3 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-09-11T05:24:52.420 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:24:52.421 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression algorithm = lz4 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression mode = aggressive 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:52.422 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:52.423 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-09-11T05:24:52.424 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-09-11T05:24:52.425 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd = 20 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: debug rbd_mirror = 15 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: log to stderr = False 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: rbd default features = 125 2024-09-11T05:24:52.426 INFO:teuthology.orchestra.run.smithi067.stdout: rbd mirror image policy type = none 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.0] 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.1] 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.2] 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.427 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.428 INFO:teuthology.orchestra.run.smithi067.stdout:[client.mirror.3] 2024-09-11T05:24:52.428 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.428 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.440 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-09-11T05:24:52.440 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-09-11T05:24:52.440 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.441 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-09-11T05:24:52.442 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-09-11T05:24:52.443 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.164 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 5000 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 53ed04a9-aee2-4881-a1db-0ad67faa19f3 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-09-11T05:24:52.444 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.445 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression algorithm = lz4 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression mode = aggressive 2024-09-11T05:24:52.446 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-09-11T05:24:52.447 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-09-11T05:24:52.448 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.449 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd = 20 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd_mirror = 15 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: log to stderr = False 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: rbd default features = 125 2024-09-11T05:24:52.450 INFO:teuthology.orchestra.run.smithi164.stdout: rbd mirror image policy type = none 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T05:24:52.451 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T05:24:52.452 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-11T05:24:52.452 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.545 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T05:24:52.545 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:52.545 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-11T05:24:52.550 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:52.550 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-11T05:24:52.597 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T05:24:52.597 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:52.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T05:24:52.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.622 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:24:52.622 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:52.670 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:24:52.671 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:52.671 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T05:24:52.671 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T05:24:52.692 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:24:52.693 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:52.796 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T05:24:52.796 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T05:24:52.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T05:24:52.842 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T05:24:52.845 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T05:24:52.845 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T05:24:52.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-11T05:24:52.880 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-11T05:24:52.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T05:24:52.926 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T05:24:52.937 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T05:24:53.012 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T05:24:53.022 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T05:24:53.094 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T05:24:53.104 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T05:24:53.176 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T05:24:53.188 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T05:24:53.264 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T05:24:53.274 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T05:24:53.350 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T05:24:53.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T05:24:53.435 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T05:24:53.445 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T05:24:53.520 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T05:24:53.530 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T05:24:53.530 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:24:53.530 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-11T05:24:53.584 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:53.584 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-11T05:24:53.584 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:24:53.584 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:53.644 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:53.645 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:54.104 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-11T05:24:54.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-11T05:24:54.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-11T05:24:54.212 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-11T05:24:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:54.221 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:54.222 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-11T05:24:54.278 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:54.279 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-11T05:24:54.279 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:24:54.279 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:24:54.345 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:54.345 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:54.345 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:54.346 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:54.819 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-11T05:24:54.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-11T05:24:54.822 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-11T05:24:54.967 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-11T05:24:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:54.979 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-11T05:24:55.032 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:24:55.032 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-11T05:24:55.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:24:55.033 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:24:55.096 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:24:55.096 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:24:55.096 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:24:55.096 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:24:55.097 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:24:55.574 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-11T05:24:55.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-11T05:24:55.576 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-11T05:24:55.713 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-11T05:24:55.724 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:24:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:24:55.725 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:55.842+0000 7fc2debfd7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-11T05:24:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:55.842+0000 7fc2debfd7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-11T05:24:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:55.846+0000 7fc2debfd7c0 -1 bdev(0x55ed89e6d800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-11T05:24:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:55.846+0000 7fc2debfd7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-11T05:24:59.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-11T05:24:59.024 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:24:59.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:59.106+0000 7fcaf1ac87c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-11T05:24:59.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:59.106+0000 7fcaf1ac87c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-11T05:24:59.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:59.106+0000 7fcaf1ac87c0 -1 bdev(0x564386e61800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-11T05:24:59.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:24:59.106+0000 7fcaf1ac87c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-11T05:25:02.268 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-11T05:25:02.325 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:25:02.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:25:02.406+0000 7f47afd427c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-11T05:25:02.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:25:02.406+0000 7f47afd427c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-11T05:25:02.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:25:02.406+0000 7f47afd427c0 -1 bdev(0x5642833ef800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-11T05:25:02.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-11T05:25:02.406+0000 7f47afd427c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-11T05:25:05.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-11T05:25:05.622 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T05:25:05.622 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:25:05.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-11T05:25:05.638 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.638 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-11T05:25:05.676 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-11T05:25:05.732 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.733 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-11T05:25:05.788 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-11T05:25:05.789 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-11T05:25:05.795 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.795 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-11T05:25:05.837 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.837 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-11T05:25:05.886 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.886 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-11T05:25:05.933 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.933 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-11T05:25:05.981 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:05.982 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-11T05:25:06.029 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:06.029 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-11T05:25:06.077 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:06.077 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-11T05:25:06.125 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:06.125 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-11T05:25:06.173 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T05:25:06.173 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-11T05:25:06.227 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCkKeFm5kRMMhAAHXxmidLiJN0lMXCJuuvfow== 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCnKeFmj1xnMhAA/DslfF6+mjARXtyZ8Ww0og== 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCrKeFmZgt6BhAANur+zCBZI3Xm+CatYCJVUg== 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCuKeFmu2VbGBAArOKzg+jiEIC3hT1bNKOG7A== 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-09-11T05:25:06.228 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCkKeFmV6KTNBAAApzbBaiTsR1XOm3A0P6iWw== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCkKeFmzjBPNxAAegxM2kJnPYuIMqdq4MCtgw== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmOPzRABAA2g0qO8LtMOcTQKt3QEIwhw== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFm+pSuBRAAgWIEiNPnnJgixQqWIoXkUQ== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFm1WKXChAA4tQh+jffvjFojoViPkOfYg== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmt9raDxAAkLGNykvMALyUlUPPF/oPAw== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.4] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFm0qjyFBAAFxwLg6eVO3DmhZcY7LIpGA== 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.5] 2024-09-11T05:25:06.229 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmDkQNGhAAn8LuQr48eFkh+RcrJoNzqg== 2024-09-11T05:25:06.230 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.6] 2024-09-11T05:25:06.230 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmlyQbHxAAI4SVXmSIrwMNy3tWb7DQWg== 2024-09-11T05:25:06.231 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:25:06.316 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:25:06.409 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:25:06.497 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:25:06.584 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:06.677 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:06.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:06.764 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:06.843 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:06.925 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:07.009 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:07.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:07.185 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:25:07.228 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T05:25:07.228 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-11T05:25:07.279 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-11T05:25:07.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-11T05:25:07.383 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T05:25:07.432 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-11T05:25:07.433 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-11T05:25:07.433 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-11T05:25:07.478 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-11T05:25:07.478 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-11T05:25:07.478 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-11T05:25:07.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-11T05:25:07.481 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-11T05:25:07.481 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T05:25:07.481 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-11T05:25:07.714 INFO:teuthology.orchestra.run.smithi164.stderr:adjusted tunables profile to default 2024-09-11T05:25:07.731 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T05:25:07.731 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-11T05:25:07.732 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:07.732 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-11T05:25:07.732 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:07.729+0000 7f3f2fa2a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T05:25:07.743 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:07.743 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-11T05:25:07.795 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:25:07.795 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-11T05:25:07.847 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new 730def7b-fac3-4e17-b5c0-bb811d132db0 0 2024-09-11T05:25:08.092 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-09-11T05:25:08.114 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new 1d736f34-58e5-4130-841c-7dbbf7ea593f 1 2024-09-11T05:25:08.119 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.117+0000 7f3f2fa2a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T05:25:08.254 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.253+0000 7f3f2fa2a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T05:25:08.315 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-09-11T05:25:08.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new 841ac8c5-41ad-496b-83b7-718ef75cc185 2 2024-09-11T05:25:08.529 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-09-11T05:25:08.546 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-11T05:25:08.547 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-11T05:25:08.549 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-11T05:25:08.549 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-11T05:25:08.549 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-11T05:25:08.550 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-11T05:25:08.551 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-11T05:25:08.551 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-11T05:25:08.553 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-11T05:25:08.553 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:08.689 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.689+0000 7f3f2fa2a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T05:25:08.729 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:08.729 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":5,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:08.529475+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:08.742 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.741+0000 7f3f2fa2a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T05:25:08.744 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-11T05:25:08.744 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T05:25:08.793 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.789+0000 7f3f2fa2a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T05:25:08.871 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.869+0000 7f3f2fa2a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T05:25:08.922 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.921+0000 7f3f2fa2a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T05:25:08.998 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:08.997+0000 7f3f2fa2a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T05:25:09.047 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:09.050 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.049+0000 7f3f2fa2a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T05:25:09.195 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.193+0000 7f3f2fa2a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T05:25:09.231 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-11T05:25:09.232 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":5,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:08.529475+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:09.246 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T05:25:09.249 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.245+0000 7f3f2fa2a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T05:25:09.475 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T05:25:09.475 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T05:25:09.475 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-09-11T05:25:09.482 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.481+0000 7f3f2fa2a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T05:25:09.602 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:25:09.602+0000 7f420bde57c0 -1 Falling back to public interface 2024-09-11T05:25:09.623 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.621+0000 7f3f2fa2a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:25:09.634 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:25:09.634+0000 7f3b0e9107c0 -1 Falling back to public interface 2024-09-11T05:25:09.698 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:25:09.698+0000 7f14eafa37c0 -1 Falling back to public interface 2024-09-11T05:25:09.736 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.733+0000 7f3f2fa2a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T05:25:09.800 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.797+0000 7f3f2fa2a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T05:25:09.939 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:09.937+0000 7f3f2fa2a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T05:25:10.440 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:10.437+0000 7f3f2fa2a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T05:25:10.516 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:10.513+0000 7f3f2fa2a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T05:25:10.666 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:10.665+0000 7f3f2fa2a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T05:25:10.742 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:10.741+0000 7f3f2fa2a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T05:25:11.038 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:11.037+0000 7f3f2fa2a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T05:25:11.095 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:11.093+0000 7f3f2fa2a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T05:25:12.131 INFO:tasks.ceph.cluster2.mgr.x.smithi067.stderr:2024-09-11T05:25:12.129+0000 7f3ed6660640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T05:25:13.663 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:25:13.662+0000 7f3b0e9107c0 -1 osd.0 0 log_to_monitors true 2024-09-11T05:25:13.755 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:25:13.754+0000 7f420bde57c0 -1 osd.1 0 log_to_monitors true 2024-09-11T05:25:13.924 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:25:13.922+0000 7f14eafa37c0 -1 osd.2 0 log_to_monitors true 2024-09-11T05:25:15.550 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:16.002 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-11T05:25:16.002 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":7,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:15.177226+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:16.023 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T05:25:16.296 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:25:16.294+0000 7f14e6f4c640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T05:25:16.325 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:25:16.326+0000 7f14ded81640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:25:16.331 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:25:16.330+0000 7f3b0b0cb640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T05:25:16.346 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:25:16.346+0000 7f4207d8e640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T05:25:16.361 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:25:16.362+0000 7f3b026ee640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:25:16.369 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:25:16.370+0000 7f41ffbc3640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:25:22.327 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:22.629 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-11T05:25:22.629 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":12,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:21.349735+0000","last_up_change":"2024-09-11T05:25:17.333638+0000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:25:19.202066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6809","nonce":2274081021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6811","nonce":2274081021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6815","nonce":2274081021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6813","nonce":2274081021}]},"public_addr":"172.21.15.164:6809/2274081021","cluster_addr":"172.21.15.164:6811/2274081021","heartbeat_back_addr":"172.21.15.164:6815/2274081021","heartbeat_front_addr":"172.21.15.164:6813/2274081021","state":["exists","up"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6801","nonce":22640096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6803","nonce":22640096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6807","nonce":22640096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6805","nonce":22640096}]},"public_addr":"172.21.15.164:6801/22640096","cluster_addr":"172.21.15.164:6803/22640096","heartbeat_back_addr":"172.21.15.164:6807/22640096","heartbeat_front_addr":"172.21.15.164:6805/22640096","state":["exists","up"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6817","nonce":3528912721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6819","nonce":3528912721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6823","nonce":3528912721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6821","nonce":3528912721}]},"public_addr":"172.21.15.164:6817/3528912721","cluster_addr":"172.21.15.164:6819/3528912721","heartbeat_back_addr":"172.21.15.164:6823/3528912721","heartbeat_front_addr":"172.21.15.164:6821/3528912721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.658323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.789994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.903487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:22.647 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T05:25:22.647 INFO:tasks.ceph:Creating RBD pool 2024-09-11T05:25:22.647 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-11T05:25:23.364 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'rbd' created 2024-09-11T05:25:23.392 DEBUG:teuthology.orchestra.run.smithi164:> rbd --cluster cluster2 pool init rbd 2024-09-11T05:25:26.397 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-11T05:25:26.397 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-11T05:25:26.397 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-11T05:25:26.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-11T05:25:26.398 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T05:25:26.736 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:26.752 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1895263611},{"type":"v1","addr":"172.21.15.67:6827","nonce":1895263611}]},"active_addr":"172.21.15.67:6827/1895263611","active_change":"2024-09-11T05:25:11.117094+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3128560990}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2311486413}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1430657689}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":190943694}]}]} 2024-09-11T05:25:26.754 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-11T05:25:26.754 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-11T05:25:26.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:27.036 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:27.036 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":16,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:26.377072+0000","last_up_change":"2024-09-11T05:25:17.333638+0000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:25:19.202066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:25:22.987864+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6809","nonce":2274081021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6811","nonce":2274081021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6815","nonce":2274081021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6813","nonce":2274081021}]},"public_addr":"172.21.15.164:6809/2274081021","cluster_addr":"172.21.15.164:6811/2274081021","heartbeat_back_addr":"172.21.15.164:6815/2274081021","heartbeat_front_addr":"172.21.15.164:6813/2274081021","state":["exists","up"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6801","nonce":22640096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6803","nonce":22640096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6807","nonce":22640096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6805","nonce":22640096}]},"public_addr":"172.21.15.164:6801/22640096","cluster_addr":"172.21.15.164:6803/22640096","heartbeat_back_addr":"172.21.15.164:6807/22640096","heartbeat_front_addr":"172.21.15.164:6805/22640096","state":["exists","up"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6817","nonce":3528912721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6819","nonce":3528912721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6823","nonce":3528912721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6821","nonce":3528912721}]},"public_addr":"172.21.15.164:6817/3528912721","cluster_addr":"172.21.15.164:6819/3528912721","heartbeat_back_addr":"172.21.15.164:6823/3528912721","heartbeat_front_addr":"172.21.15.164:6821/3528912721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.658323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.789994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.903487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:27.053 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-11T05:25:27.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T05:25:27.346 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:27.346 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":16,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T05:25:26.377072+0000","last_up_change":"2024-09-11T05:25:17.333638+0000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:25:19.202066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:25:22.987864+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6809","nonce":2274081021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6811","nonce":2274081021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6815","nonce":2274081021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6813","nonce":2274081021}]},"public_addr":"172.21.15.164:6809/2274081021","cluster_addr":"172.21.15.164:6811/2274081021","heartbeat_back_addr":"172.21.15.164:6815/2274081021","heartbeat_front_addr":"172.21.15.164:6813/2274081021","state":["exists","up"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6801","nonce":22640096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6803","nonce":22640096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6807","nonce":22640096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6805","nonce":22640096}]},"public_addr":"172.21.15.164:6801/22640096","cluster_addr":"172.21.15.164:6803/22640096","heartbeat_back_addr":"172.21.15.164:6807/22640096","heartbeat_front_addr":"172.21.15.164:6805/22640096","state":["exists","up"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6817","nonce":3528912721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6819","nonce":3528912721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6823","nonce":3528912721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6821","nonce":3528912721}]},"public_addr":"172.21.15.164:6817/3528912721","cluster_addr":"172.21.15.164:6819/3528912721","heartbeat_back_addr":"172.21.15.164:6823/3528912721","heartbeat_front_addr":"172.21.15.164:6821/3528912721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.658323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.789994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.903487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:25:27.363 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-11T05:25:27.364 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-11T05:25:27.364 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-11T05:25:27.513 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2024-09-11T05:25:27.514 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T05:25:27.531 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2024-09-11T05:25:27.532 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T05:25:27.534 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2024-09-11T05:25:27.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T05:25:27.804 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2024-09-11T05:25:27.821 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-11T05:25:27.850 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2024-09-11T05:25:27.867 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-09-11T05:25:27.868 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2024-09-11T05:25:27.885 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-11T05:25:28.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T05:25:28.868 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T05:25:28.886 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T05:25:29.160 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2024-09-11T05:25:29.177 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-11T05:25:29.193 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2024-09-11T05:25:29.193 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2024-09-11T05:25:29.212 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-11T05:25:29.213 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-09-11T05:25:30.178 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T05:25:30.213 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T05:25:30.213 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T05:25:30.502 INFO:teuthology.orchestra.run.smithi164.stdout:34359738375 2024-09-11T05:25:30.518 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.0 2024-09-11T05:25:30.519 DEBUG:teuthology.parallel:result is None 2024-09-11T05:25:30.582 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2024-09-11T05:25:30.591 INFO:teuthology.orchestra.run.smithi164.stdout:34359738375 2024-09-11T05:25:30.600 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.2 2024-09-11T05:25:30.600 DEBUG:teuthology.parallel:result is None 2024-09-11T05:25:30.609 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.1 2024-09-11T05:25:30.609 DEBUG:teuthology.parallel:result is None 2024-09-11T05:25:30.609 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T05:25:30.610 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T05:25:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:30.923 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T05:25:30.940 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-11T05:25:29.128207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81620,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033900,"statfs":{"total":289910292480,"available":289826713600,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.759964"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390971+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T05:25:26.390971+0000","last_peered":"2024-09-11T05:25:26.390971+0000","last_clean":"2024-09-11T05:25:26.390971+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T05:25:26.390971+0000","last_undegraded":"2024-09-11T05:25:26.390971+0000","last_fullsized":"2024-09-11T05:25:26.390971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390354+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T05:25:26.390354+0000","last_peered":"2024-09-11T05:25:26.390354+0000","last_clean":"2024-09-11T05:25:26.390354+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T05:25:26.390354+0000","last_undegraded":"2024-09-11T05:25:26.390354+0000","last_fullsized":"2024-09-11T05:25:26.390354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390055+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T05:25:26.390055+0000","last_peered":"2024-09-11T05:25:26.390055+0000","last_clean":"2024-09-11T05:25:26.390055+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T05:25:26.390055+0000","last_undegraded":"2024-09-11T05:25:26.390055+0000","last_fullsized":"2024-09-11T05:25:26.390055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.389660+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T05:25:26.389660+0000","last_peered":"2024-09-11T05:25:26.389660+0000","last_clean":"2024-09-11T05:25:26.389660+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T05:25:26.389660+0000","last_undegraded":"2024-09-11T05:25:26.389660+0000","last_fullsized":"2024-09-11T05:25:26.389660+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.386876+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T05:25:26.386876+0000","last_peered":"2024-09-11T05:25:26.386876+0000","last_clean":"2024-09-11T05:25:26.386876+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T05:25:26.386876+0000","last_undegraded":"2024-09-11T05:25:26.386876+0000","last_fullsized":"2024-09-11T05:25:26.386876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:55.262608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.878948+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T05:25:26.878948+0000","last_peered":"2024-09-11T05:25:26.878948+0000","last_clean":"2024-09-11T05:25:26.878948+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T05:25:26.878948+0000","last_undegraded":"2024-09-11T05:25:26.878948+0000","last_fullsized":"2024-09-11T05:25:26.878948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.878938+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T05:25:26.878938+0000","last_peered":"2024-09-11T05:25:26.878938+0000","last_clean":"2024-09-11T05:25:26.878938+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T05:25:26.878938+0000","last_undegraded":"2024-09-11T05:25:26.878938+0000","last_fullsized":"2024-09-11T05:25:26.878938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.389715+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T05:25:26.389715+0000","last_peered":"2024-09-11T05:25:26.389715+0000","last_clean":"2024-09-11T05:25:26.389715+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T05:25:26.389715+0000","last_undegraded":"2024-09-11T05:25:26.389715+0000","last_fullsized":"2024-09-11T05:25:26.389715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.387508+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T05:25:26.387508+0000","last_peered":"2024-09-11T05:25:26.387508+0000","last_clean":"2024-09-11T05:25:26.387508+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T05:25:26.387508+0000","last_undegraded":"2024-09-11T05:25:26.387508+0000","last_fullsized":"2024-09-11T05:25:26.387508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:25:30.941 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T05:25:31.207 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:31.207 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T05:25:31.223 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T05:25:31.128881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81620,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033900,"statfs":{"total":289910292480,"available":289826713600,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.761113"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390971+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T05:25:26.390971+0000","last_peered":"2024-09-11T05:25:26.390971+0000","last_clean":"2024-09-11T05:25:26.390971+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T05:25:26.390971+0000","last_undegraded":"2024-09-11T05:25:26.390971+0000","last_fullsized":"2024-09-11T05:25:26.390971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390354+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T05:25:26.390354+0000","last_peered":"2024-09-11T05:25:26.390354+0000","last_clean":"2024-09-11T05:25:26.390354+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T05:25:26.390354+0000","last_undegraded":"2024-09-11T05:25:26.390354+0000","last_fullsized":"2024-09-11T05:25:26.390354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.390055+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T05:25:26.390055+0000","last_peered":"2024-09-11T05:25:26.390055+0000","last_clean":"2024-09-11T05:25:26.390055+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T05:25:26.390055+0000","last_undegraded":"2024-09-11T05:25:26.390055+0000","last_fullsized":"2024-09-11T05:25:26.390055+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.389660+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T05:25:26.389660+0000","last_peered":"2024-09-11T05:25:26.389660+0000","last_clean":"2024-09-11T05:25:26.389660+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T05:25:26.389660+0000","last_undegraded":"2024-09-11T05:25:26.389660+0000","last_fullsized":"2024-09-11T05:25:26.389660+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.386876+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T05:25:26.386876+0000","last_peered":"2024-09-11T05:25:26.386876+0000","last_clean":"2024-09-11T05:25:26.386876+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T05:25:26.386876+0000","last_undegraded":"2024-09-11T05:25:26.386876+0000","last_fullsized":"2024-09-11T05:25:26.386876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:55.262608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.878948+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T05:25:26.878948+0000","last_peered":"2024-09-11T05:25:26.878948+0000","last_clean":"2024-09-11T05:25:26.878948+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T05:25:26.878948+0000","last_undegraded":"2024-09-11T05:25:26.878948+0000","last_fullsized":"2024-09-11T05:25:26.878948+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.878938+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T05:25:26.878938+0000","last_peered":"2024-09-11T05:25:26.878938+0000","last_clean":"2024-09-11T05:25:26.878938+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T05:25:26.878938+0000","last_undegraded":"2024-09-11T05:25:26.878938+0000","last_fullsized":"2024-09-11T05:25:26.878938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.389715+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T05:25:26.389715+0000","last_peered":"2024-09-11T05:25:26.389715+0000","last_clean":"2024-09-11T05:25:26.389715+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T05:25:26.389715+0000","last_undegraded":"2024-09-11T05:25:26.389715+0000","last_fullsized":"2024-09-11T05:25:26.389715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T05:25:26.387508+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T05:25:26.387508+0000","last_peered":"2024-09-11T05:25:26.387508+0000","last_clean":"2024-09-11T05:25:26.387508+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T05:25:26.387508+0000","last_undegraded":"2024-09-11T05:25:26.387508+0000","last_fullsized":"2024-09-11T05:25:26.387508+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:25:31.224 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T05:25:31.224 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-11T05:25:31.224 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-11T05:25:31.224 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-11T05:25:31.553 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T05:25:31.553 INFO:teuthology.orchestra.run.smithi164.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T05:25:31.572 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-11T05:25:31.572 INFO:teuthology.run_tasks:Running task exec... 2024-09-11T05:25:31.582 INFO:teuthology.task.exec:Executing custom commands... 2024-09-11T05:25:31.582 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T05:25:31.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:31.944 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2024-09-11T05:25:31.945 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB9KeFmuVp5LhAAZEy88J9f18FBYw1y+/9WlA== 2024-09-11T05:25:31.945 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T05:25:31.945 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror 2024-09-11T05:25:31.962 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:32.323 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2024-09-11T05:25:32.323 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB9KeFmt8YyNBAA+FYVbM07tFO8nyXn+0Ya0Q== 2024-09-11T05:25:32.323 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:32.323 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:32.323 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.0 2024-09-11T05:25:32.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:32.720 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2024-09-11T05:25:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB9KeFmsGMLORAAEOcrnekcUxgik6YRN6R1fA== 2024-09-11T05:25:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.1 2024-09-11T05:25:32.738 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:33.090 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2024-09-11T05:25:33.090 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB+KeFmqoCNAhAAy2RjZHiu7AlUvx3t0u8Ouw== 2024-09-11T05:25:33.090 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:33.090 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:33.091 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.2 2024-09-11T05:25:33.108 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:33.457 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2024-09-11T05:25:33.457 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQB+KeFmE7SoBxAAwKJO4TTkUCritMyYKBg3aw== 2024-09-11T05:25:33.457 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:33.457 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:33.457 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.3 2024-09-11T05:25:33.475 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:33.813 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2024-09-11T05:25:33.814 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCkKeFmzjBPNxAAegxM2kJnPYuIMqdq4MCtgw== 2024-09-11T05:25:33.814 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T05:25:33.814 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:33.814 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror 2024-09-11T05:25:33.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:34.184 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2024-09-11T05:25:34.184 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmOPzRABAA2g0qO8LtMOcTQKt3QEIwhw== 2024-09-11T05:25:34.184 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:34.184 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:34.184 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.0 2024-09-11T05:25:34.201 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:34.543 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2024-09-11T05:25:34.543 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFm+pSuBRAAgWIEiNPnnJgixQqWIoXkUQ== 2024-09-11T05:25:34.543 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:34.544 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:34.544 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.1 2024-09-11T05:25:34.561 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:34.925 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2024-09-11T05:25:34.925 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFm1WKXChAA4tQh+jffvjFojoViPkOfYg== 2024-09-11T05:25:34.925 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:34.925 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.2 2024-09-11T05:25:34.944 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T05:25:35.316 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2024-09-11T05:25:35.316 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQClKeFmt9raDxAAkLGNykvMALyUlUPPF/oPAw== 2024-09-11T05:25:35.316 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2024-09-11T05:25:35.316 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2024-09-11T05:25:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.3 2024-09-11T05:25:35.335 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T05:25:35.371 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T05:25:35.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T05:25:35.374 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T05:25:35.382 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T05:25:35.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T05:25:35.385 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:25:35.395 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:25:35.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:25:35.397 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T05:25:35.406 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T05:25:35.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T05:25:35.408 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:25:35.418 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:25:35.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:25:35.420 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T05:25:35.428 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T05:25:35.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T05:25:35.431 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T05:25:35.440 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T05:25:35.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T05:25:35.442 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T05:25:35.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:25:35.451 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:25:35.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:25:35.454 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-11T05:25:35.478 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726032335 2024-09-11T05:25:35.478 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-11T05:25:35.478 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-11T05:25:35.488 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726032335 2024-09-11T05:25:35.489 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-11T05:25:35.489 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T05:25:35.502 INFO:tasks.workunit:Pulling workunits from ref daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T05:25:35.503 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T05:25:35.503 DEBUG:teuthology.orchestra.run.smithi164:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T05:25:35.504 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-11T05:25:35.504 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-11T05:25:35.504 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-11T05:25:35.504 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-11T05:25:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:25:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-11T05:25:35.509 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T05:25:35.556 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T05:25:35.556 DEBUG:teuthology.orchestra.run.smithi164:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-11T05:25:35.604 INFO:tasks.workunit:timeout=6h 2024-09-11T05:25:35.604 INFO:tasks.workunit:cleanup=True 2024-09-11T05:25:35.605 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T05:25:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Note: switching to 'daaf78de334647fc0c800f4ac495cc9475a26bae'. 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: git switch -c 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Or undo this operation with: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: git switch - 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:26:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:HEAD is now at daaf78de334 Merge pull request #59682 from zdover23/wip-doc-2024-09-10-backport-59677-to-squid 2024-09-11T05:26:29.721 DEBUG:teuthology.orchestra.run.smithi164:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-11T05:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T05:26:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T05:26:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T05:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T05:26:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T05:26:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T05:26:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T05:26:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T05:26:30.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T05:26:30.718 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-11T05:26:30.718 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-11T05:26:30.770 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-11T05:26:30.771 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-11T05:26:30.772 DEBUG:teuthology.orchestra.run.smithi164:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daaf78de334647fc0c800f4ac495cc9475a26bae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T05:26:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ IMAGE_COUNT=50 2024-09-11T05:26:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ export LOCKDEP=0 2024-09-11T05:26:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ LOCKDEP=0 2024-09-11T05:26:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ type xmlstarlet 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ XMLSTARLET=xmlstarlet 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CLUSTER1=cluster1 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CLUSTER2=cluster2 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-11T05:26:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ POOL=mirror 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ PARENT_POOL=mirror_parent 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ NS1=ns1 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ NS2=ns2 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ TEMPDIR= 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ID=mirror 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ MIRROR_POOL_MODE=image 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' snapshot = snapshot ']' 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ MIRROR_POOL_MODE=image 2024-09-11T05:26:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-11T05:26:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS='--id mirror' 2024-09-11T05:26:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-11T05:26:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T05:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-11T05:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-11T05:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_BIN=. 2024-09-11T05:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_SRC=. 2024-09-11T05:26:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-11T05:26:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 -gt 0 ']' 2024-09-11T05:26:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup 2024-09-11T05:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local c 2024-09-11T05:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-11T05:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_tempdir 2024-09-11T05:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mktemp -d 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ TEMPDIR=/tmp/tmp.Ange9SOiZn 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z 1 ']' 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_pools cluster1 cluster2 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_map_file 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_addr 2024-09-11T05:26:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local admin_key_file 2024-09-11T05:26:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid 2024-09-11T05:26:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-11T05:26:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' already exists 2024-09-11T05:26:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-11T05:26:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' already exists 2024-09-11T05:26:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-11T05:26:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-11T05:26:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-11T05:26:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-11T05:26:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-11T05:26:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-11T05:26:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-11T05:26:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z ']' 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z ']' 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster1 mirror cluster2 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster2 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:26:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=af3f2d07-6a87-4d52-867d-2bc239053a8b 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2024-09-11T05:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror_parent 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster2 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2024-09-11T05:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2024-09-11T05:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T05:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-11T05:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=013c3ac5-7fa1-40ad-903d-fa5694bac6c0 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_pools cluster2 cluster1 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_map_file 2024-09-11T05:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_addr 2024-09-11T05:26:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local admin_key_file 2024-09-11T05:26:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid 2024-09-11T05:26:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-11T05:26:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' already exists 2024-09-11T05:26:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-11T05:26:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' already exists 2024-09-11T05:26:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-11T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-11T05:26:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T05:26:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-11T05:26:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-11T05:26:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-11T05:26:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-11T05:26:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-11T05:26:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-11T05:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z ']' 2024-09-11T05:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z ']' 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster2 mirror cluster1 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster1 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T05:26:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:26:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-11T05:26:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=f47c6719-5099-4349-a7c0-e7564c6a2880 2024-09-11T05:26:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T05:26:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:26:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T05:26:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2024-09-11T05:26:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror_parent 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster1 2024-09-11T05:26:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:26:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=bc7f0b39-a3a4-4dee-b2f3-646c5e189097 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:26:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:26:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirrors cluster1 2024-09-11T05:26:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 0 3 2024-09-11T05:26:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:0 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2024-09-11T05:26:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:2 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:3 2024-09-11T05:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirrors cluster2 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 0 3 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:0 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:0 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:0 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2024-09-11T05:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster2 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:1 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:1 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:1 2024-09-11T05:26:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster2 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:2 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:2 2024-09-11T05:26:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:3 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:3 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:3 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster2 2024-09-11T05:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:26:47 TEST: add image and test replay after client crashes 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:26:47 TEST: add image and test replay after client crashes 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=test 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror test 512M 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:26:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:26:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:26:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:26:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-11T05:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-11T05:26:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:26:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-11T05:26:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:26:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:26:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10aaa3862161 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10aaa3862161 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:26:47 2024 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:26:47 2024 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:26:47 2024 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:26:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:26:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:26:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:26:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:26:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:26:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:26:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:26:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:26:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:26:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:26:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:26:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:26:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:26:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:26:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:26:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:26:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:26:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:26:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:3 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:3 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=3 2024-09-11T05:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T05:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:3 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:3 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=3 2024-09-11T05:26:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=3 2024-09-11T05:26:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T05:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T05:26:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 1 2024-09-11T05:26:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:26:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:26:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:26:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:26:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:26:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:26:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:26:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:26:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:26:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:26:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:26:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:26:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:26:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:26:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:3 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:3 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=3 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:3 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:3 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=3 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=3 2024-09-11T05:26:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T05:26:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T05:26:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 2 2024-09-11T05:26:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:26:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:26:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:26:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:26:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:26:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:26:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:26:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:26:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:26:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:26:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:26:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:26:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:26:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:26:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:26:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:26:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:26:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:26:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:26:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T05:26:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:3 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:3 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=3 2024-09-11T05:26:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T05:26:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:3 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:3 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=3 2024-09-11T05:26:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=3 2024-09-11T05:26:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T05:26:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T05:26:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 4 2024-09-11T05:26:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:26:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:26:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:26:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:26:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:26:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:26:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:26:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:26:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:26:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:26:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:26:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:26:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:26:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:26:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:26:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:26:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:26:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:26:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:26:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:26:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:26:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:3 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:3 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:26:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=3 2024-09-11T05:26:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:3 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T05:26:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:3 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=3 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=3 2024-09-11T05:26:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T05:26:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:26:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T05:26:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:26:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:26:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:26:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:26:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 8 2024-09-11T05:27:04.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7776864879427355) > weight (0.25) 2024-09-11T05:27:04.506 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9411407158439674) > weight (0.25) 2024-09-11T05:27:04.506 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-11T05:27:04.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-11T05:27:04.507 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:27:04.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T05:27:04.503+0000 7f2e4d56e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 28954) UID: 1000 2024-09-11T05:27:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:27:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:27:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:27:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:27:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:27:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:27:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:27:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:27:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:27:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:27:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:27:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:27:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:27:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:27:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:27:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:3 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:3 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=3 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T05:27:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=3 2024-09-11T05:27:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:3 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:3 2024-09-11T05:27:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=3 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=3 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=3 2024-09-11T05:27:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T05:27:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:27:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:27:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T05:27:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T05:27:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 8 2024-09-11T05:27:10.508 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-11T05:27:10.509 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:27:10.509 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-11T05:27:10.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T05:27:10.509 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T05:27:10.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T05:27:10.513 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-11T05:27:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:27:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:27:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:27:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:27:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:27:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:27:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:27:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:27:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:27:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:27:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:27:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:27:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:27:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:27:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:27:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:27:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:27:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name= 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 10 2024-09-11T05:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:27:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:27:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:27:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:27:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 16121856 bytes 2024-09-11T05:27:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 49123328 bytes 2024-09-11T05:27:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 66762752 bytes 2024-09-11T05:27:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 125917184 bytes 2024-09-11T05:27:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 140549120 bytes 2024-09-11T05:27:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 223057920 bytes 2024-09-11T05:27:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 259165184 bytes 2024-09-11T05:27:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 279023616 bytes 2024-09-11T05:27:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 392977408 bytes 2024-09-11T05:27:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 410421248 bytes 2024-09-11T05:27:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 411468800 bytes 2024-09-11T05:27:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 436284416 bytes 2024-09-11T05:27:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 496484352 bytes 2024-09-11T05:27:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 529175552 bytes 2024-09-11T05:27:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 548022272 bytes 2024-09-11T05:27:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 612576256 bytes 2024-09-11T05:27:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 643620864 bytes 2024-09-11T05:27:16.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7867192667565887) > weight (0.25) 2024-09-11T05:27:16.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.396393911330128) > weight (0.25) 2024-09-11T05:27:16.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3218474874376964) > weight (0.25) 2024-09-11T05:27:16.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9994734950617241) > weight (0.25) 2024-09-11T05:27:16.505 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-09-11T05:27:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 696795136 bytes 2024-09-11T05:27:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 732690432 bytes 2024-09-11T05:27:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 748579840 bytes 2024-09-11T05:27:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 782028800 bytes 2024-09-11T05:27:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 818264064 bytes 2024-09-11T05:27:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 828026880 bytes 2024-09-11T05:27:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 849226752 bytes 2024-09-11T05:27:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 896008192 bytes 2024-09-11T05:27:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 931601408 bytes 2024-09-11T05:27:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 964040704 bytes 2024-09-11T05:27:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1032492032 bytes 2024-09-11T05:27:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1071119360 bytes 2024-09-11T05:27:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1141454848 bytes 2024-09-11T05:27:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1187726336 bytes 2024-09-11T05:27:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1222567936 bytes 2024-09-11T05:27:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1291692032 bytes 2024-09-11T05:27:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1329968128 bytes 2024-09-11T05:27:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1413990400 bytes 2024-09-11T05:27:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1484478464 bytes 2024-09-11T05:27:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1514894336 bytes 2024-09-11T05:27:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1565704192 bytes 2024-09-11T05:27:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1604919296 bytes 2024-09-11T05:27:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1664968704 bytes 2024-09-11T05:27:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1738114048 bytes 2024-09-11T05:27:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1779428352 bytes 2024-09-11T05:27:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1789064192 bytes 2024-09-11T05:27:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1892165632 bytes 2024-09-11T05:27:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1942946816 bytes 2024-09-11T05:27:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1984577536 bytes 2024-09-11T05:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2032261120 bytes 2024-09-11T05:27:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2054548480 bytes 2024-09-11T05:27:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2095758336 bytes 2024-09-11T05:27:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2147678208 bytes 2024-09-11T05:27:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2207289344 bytes 2024-09-11T05:27:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2215330816 bytes 2024-09-11T05:27:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2256117760 bytes 2024-09-11T05:27:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2269773824 bytes 2024-09-11T05:27:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2290434048 bytes 2024-09-11T05:27:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2339312640 bytes 2024-09-11T05:27:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2402662400 bytes 2024-09-11T05:27:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2451075072 bytes 2024-09-11T05:27:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2521066496 bytes 2024-09-11T05:27:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2535609344 bytes 2024-09-11T05:27:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2583918592 bytes 2024-09-11T05:27:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2658564096 bytes 2024-09-11T05:27:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2659181568 bytes 2024-09-11T05:27:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2659812352 bytes 2024-09-11T05:27:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2707593216 bytes 2024-09-11T05:27:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2761678848 bytes 2024-09-11T05:27:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2784960512 bytes 2024-09-11T05:27:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2847767552 bytes 2024-09-11T05:27:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2872292352 bytes 2024-09-11T05:27:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2891573248 bytes 2024-09-11T05:27:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2958248960 bytes 2024-09-11T05:27:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3062948864 bytes 2024-09-11T05:27:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3119834112 bytes 2024-09-11T05:27:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3143678976 bytes 2024-09-11T05:27:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3224905728 bytes 2024-09-11T05:27:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3297694720 bytes 2024-09-11T05:27:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3370720256 bytes 2024-09-11T05:27:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3524357120 bytes 2024-09-11T05:27:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3588127744 bytes 2024-09-11T05:27:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3637741568 bytes 2024-09-11T05:27:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3676463104 bytes 2024-09-11T05:27:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3691425792 bytes 2024-09-11T05:27:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3722721280 bytes 2024-09-11T05:27:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3740572672 bytes 2024-09-11T05:27:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3741201408 bytes 2024-09-11T05:27:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3807050752 bytes 2024-09-11T05:27:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3827034112 bytes 2024-09-11T05:27:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3848019968 bytes 2024-09-11T05:27:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3901460480 bytes 2024-09-11T05:27:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3984902144 bytes 2024-09-11T05:27:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 4006080512 bytes 2024-09-11T05:27:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4082396160 bytes 2024-09-11T05:27:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4122910720 bytes 2024-09-11T05:27:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 4159286272 bytes 2024-09-11T05:27:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4303651840 bytes 2024-09-11T05:27:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4352492544 bytes 2024-09-11T05:27:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4407780352 bytes 2024-09-11T05:27:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4456650752 bytes 2024-09-11T05:27:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4494121984 bytes 2024-09-11T05:27:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4547281920 bytes 2024-09-11T05:27:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4560072704 bytes 2024-09-11T05:27:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4599735296 bytes 2024-09-11T05:27:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4665370624 bytes 2024-09-11T05:27:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4711370752 bytes 2024-09-11T05:27:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4742898688 bytes 2024-09-11T05:27:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4795688960 bytes 2024-09-11T05:27:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4811312128 bytes 2024-09-11T05:27:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10800 IOs, 4834004992 bytes 2024-09-11T05:27:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10900 IOs, 4876151808 bytes 2024-09-11T05:27:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11000 IOs, 4893673472 bytes 2024-09-11T05:27:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11100 IOs, 4921402368 bytes 2024-09-11T05:27:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11200 IOs, 4967008256 bytes 2024-09-11T05:27:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11300 IOs, 5004666880 bytes 2024-09-11T05:27:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11400 IOs, 5021820928 bytes 2024-09-11T05:27:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11500 IOs, 5038940160 bytes 2024-09-11T05:27:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11600 IOs, 5063819264 bytes 2024-09-11T05:27:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11700 IOs, 5072705536 bytes 2024-09-11T05:27:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:27:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:27:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:27:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:27:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:27:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:27:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: ' 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2024-09-11T05:27:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2024-09-11T05:27:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:27:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:27:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: ' 2024-09-11T05:27:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:27:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2024-09-11T05:27:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2024-09-11T05:27:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:27:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:27:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:27:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:27:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: ' 2024-09-11T05:27:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:27:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2024-09-11T05:27:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2024-09-11T05:27:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:27:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:27:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:27:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:27:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032407,"remote_snapshot_timestamp":1726032407,"replay_state":"idle"} 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:27:35 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:27:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:27:35' 2024-09-11T05:27:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:27:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:27:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:27:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032407,"remote_snapshot_timestamp":1726032407,"replay_state":"idle"} 2024-09-11T05:27:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:27:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:27:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:27:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:27:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:27:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:27:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:27:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:27:35' 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:27:35' 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:27:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:27:35' 2024-09-11T05:27:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:27:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:27:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:27:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142084262mirror.0smithi16419.1.1-298-gb53b6d3efalseOK42104261mirror.1smithi16419.1.1-298-gb53b6d3efalseOK42164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032407,"remote_snapshot_timestamp":1726032407,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:27:35cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:27:35' 2024-09-11T05:27:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:27:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:27:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:27:35' 2024-09-11T05:27:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:27:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:27:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:27:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:27:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:27:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap1 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2024-09-11T05:27:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:27:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:27:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:27:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:27:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:27:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:27:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:27:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:27:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:27:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:27:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:27:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:27:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:27:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:27:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:27:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:27:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:27:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:27:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:27:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:27:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:27:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:27:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:27:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:27:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:27:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:27:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:27:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:27:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:27:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:27:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:27:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:27:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:27:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:27:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:27:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:27:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:27:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:27:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:27:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:27:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:27:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:27:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:27:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:27:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:27:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:27:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:27:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:27:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:27:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 5 2024-09-11T05:27:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:27:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:27:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:27:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:27:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:27:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:27:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:27:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:27:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=5 2024-09-11T05:27:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:27:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:27:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:28:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=3 2024-09-11T05:28:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 5 = 3 2024-09-11T05:28:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:28:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:28:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:28:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:28:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:28:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 5 = '' 2024-09-11T05:28:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:28:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 5 = '' 2024-09-11T05:28:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=5 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 5 = 5 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2024-09-11T05:28:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:28:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:28:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:28:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-11T05:28:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:28:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:28:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:28:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:28:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:28:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:28:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:28:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap1-clean 2024-09-11T05:28:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-11T05:28:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2024-09-11T05:28:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:28:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:28:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:28:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:28:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:28:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:28:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:28:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 618496 bytes 2024-09-11T05:28:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 54265856 bytes 2024-09-11T05:28:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 84661248 bytes 2024-09-11T05:28:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 122976256 bytes 2024-09-11T05:28:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 139532288 bytes 2024-09-11T05:28:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 190704640 bytes 2024-09-11T05:28:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 216058880 bytes 2024-09-11T05:28:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 273446912 bytes 2024-09-11T05:28:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 303206400 bytes 2024-09-11T05:28:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 334543872 bytes 2024-09-11T05:28:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 337939456 bytes 2024-09-11T05:28:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 364788736 bytes 2024-09-11T05:28:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 384393216 bytes 2024-09-11T05:28:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 391022592 bytes 2024-09-11T05:28:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 409293824 bytes 2024-09-11T05:28:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 423733248 bytes 2024-09-11T05:28:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 428584960 bytes 2024-09-11T05:28:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 509730816 bytes 2024-09-11T05:28:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 551057408 bytes 2024-09-11T05:28:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 630646784 bytes 2024-09-11T05:28:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 692802560 bytes 2024-09-11T05:28:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 737394688 bytes 2024-09-11T05:28:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 756487168 bytes 2024-09-11T05:28:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 787651584 bytes 2024-09-11T05:28:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 849918976 bytes 2024-09-11T05:28:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 920835072 bytes 2024-09-11T05:28:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 987054080 bytes 2024-09-11T05:28:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1002722304 bytes 2024-09-11T05:28:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1057912832 bytes 2024-09-11T05:28:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1070757888 bytes 2024-09-11T05:28:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1149497344 bytes 2024-09-11T05:28:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1161046016 bytes 2024-09-11T05:28:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1203543040 bytes 2024-09-11T05:28:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1246836736 bytes 2024-09-11T05:28:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1278215168 bytes 2024-09-11T05:28:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1308801024 bytes 2024-09-11T05:28:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1309561856 bytes 2024-09-11T05:28:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1341313024 bytes 2024-09-11T05:28:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1357103104 bytes 2024-09-11T05:28:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1386518528 bytes 2024-09-11T05:28:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1409150976 bytes 2024-09-11T05:28:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1528029184 bytes 2024-09-11T05:28:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1556781056 bytes 2024-09-11T05:28:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1577609216 bytes 2024-09-11T05:28:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1614392320 bytes 2024-09-11T05:28:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1660041216 bytes 2024-09-11T05:28:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1698661376 bytes 2024-09-11T05:28:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1774259200 bytes 2024-09-11T05:28:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1796019200 bytes 2024-09-11T05:28:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1867693056 bytes 2024-09-11T05:28:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1877772288 bytes 2024-09-11T05:28:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1910580224 bytes 2024-09-11T05:28:12.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1982085120 bytes 2024-09-11T05:28:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2030681088 bytes 2024-09-11T05:28:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2102052864 bytes 2024-09-11T05:28:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2199979008 bytes 2024-09-11T05:28:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2219829248 bytes 2024-09-11T05:28:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2236366848 bytes 2024-09-11T05:28:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2276756480 bytes 2024-09-11T05:28:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2291369984 bytes 2024-09-11T05:28:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2310355968 bytes 2024-09-11T05:28:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2344098816 bytes 2024-09-11T05:28:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2394336256 bytes 2024-09-11T05:28:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2441779200 bytes 2024-09-11T05:28:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2499137536 bytes 2024-09-11T05:28:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2559268864 bytes 2024-09-11T05:28:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2618464256 bytes 2024-09-11T05:28:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2661745664 bytes 2024-09-11T05:28:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2707323904 bytes 2024-09-11T05:28:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2767310848 bytes 2024-09-11T05:28:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2817932288 bytes 2024-09-11T05:28:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2877276160 bytes 2024-09-11T05:28:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2905985024 bytes 2024-09-11T05:28:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2960668672 bytes 2024-09-11T05:28:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3039291392 bytes 2024-09-11T05:28:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3087132672 bytes 2024-09-11T05:28:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3114983424 bytes 2024-09-11T05:28:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3137971200 bytes 2024-09-11T05:28:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3174086656 bytes 2024-09-11T05:28:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3174715392 bytes 2024-09-11T05:28:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3215096832 bytes 2024-09-11T05:28:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3276585984 bytes 2024-09-11T05:28:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3335523328 bytes 2024-09-11T05:28:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3412968448 bytes 2024-09-11T05:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3416587264 bytes 2024-09-11T05:28:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3534478336 bytes 2024-09-11T05:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3582056448 bytes 2024-09-11T05:28:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3619885056 bytes 2024-09-11T05:28:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3661434880 bytes 2024-09-11T05:28:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3707845632 bytes 2024-09-11T05:28:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3767017472 bytes 2024-09-11T05:28:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3791470592 bytes 2024-09-11T05:28:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3847384064 bytes 2024-09-11T05:28:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3870879744 bytes 2024-09-11T05:28:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3897521152 bytes 2024-09-11T05:28:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3910618112 bytes 2024-09-11T05:28:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 3940280320 bytes 2024-09-11T05:28:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 3981361152 bytes 2024-09-11T05:28:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4047307776 bytes 2024-09-11T05:28:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4117618688 bytes 2024-09-11T05:28:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4186309632 bytes 2024-09-11T05:28:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4244229120 bytes 2024-09-11T05:28:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4303639552 bytes 2024-09-11T05:28:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4324183040 bytes 2024-09-11T05:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4441361408 bytes 2024-09-11T05:28:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4522301440 bytes 2024-09-11T05:28:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4546814976 bytes 2024-09-11T05:28:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10800 IOs, 4573577216 bytes 2024-09-11T05:28:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10900 IOs, 4594770944 bytes 2024-09-11T05:28:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11000 IOs, 4636685312 bytes 2024-09-11T05:28:18.514 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:28:18.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:28:18.514 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-11T05:28:18.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:28:18.510+0000 7fc3ffeb0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 28922) UID: 1000 2024-09-11T05:28:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11100 IOs, 4707249152 bytes 2024-09-11T05:28:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11200 IOs, 4741830656 bytes 2024-09-11T05:28:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11300 IOs, 4754100224 bytes 2024-09-11T05:28:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11400 IOs, 4799639552 bytes 2024-09-11T05:28:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11500 IOs, 4821031936 bytes 2024-09-11T05:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11600 IOs, 4851369984 bytes 2024-09-11T05:28:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11700 IOs, 4957928448 bytes 2024-09-11T05:28:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11800 IOs, 4992404480 bytes 2024-09-11T05:28:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11900 IOs, 5008780288 bytes 2024-09-11T05:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12000 IOs, 5079169024 bytes 2024-09-11T05:28:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12100 IOs, 5079784448 bytes 2024-09-11T05:28:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12200 IOs, 5138953216 bytes 2024-09-11T05:28:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12300 IOs, 5201240064 bytes 2024-09-11T05:28:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12400 IOs, 5224014848 bytes 2024-09-11T05:28:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12500 IOs, 5337879552 bytes 2024-09-11T05:28:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12600 IOs, 5358398464 bytes 2024-09-11T05:28:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12700 IOs, 5386755072 bytes 2024-09-11T05:28:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12800 IOs, 5440661504 bytes 2024-09-11T05:28:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12900 IOs, 5441351680 bytes 2024-09-11T05:28:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13000 IOs, 5464216576 bytes 2024-09-11T05:28:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13100 IOs, 5486761984 bytes 2024-09-11T05:28:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13200 IOs, 5501284352 bytes 2024-09-11T05:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13300 IOs, 5506264064 bytes 2024-09-11T05:28:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13400 IOs, 5539833856 bytes 2024-09-11T05:28:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13500 IOs, 5577977856 bytes 2024-09-11T05:28:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13600 IOs, 5599700992 bytes 2024-09-11T05:28:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:13700 IOs, 5628362752 bytes 2024-09-11T05:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:28:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:28:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:28:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:28:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:28:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:28:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:28:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":15938355.2,"bytes_per_snapshot":267946496.0,"last_snapshot_bytes":535892992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032479,"remote_snapshot_timestamp":1726032479,"replay_state":"idle"} 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:28:05 2024-09-11T05:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:28:05' 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:28:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":15938355.2,"bytes_per_snapshot":267946496.0,"last_snapshot_bytes":535892992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032479,"remote_snapshot_timestamp":1726032479,"replay_state":"idle"} 2024-09-11T05:28:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:28:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:28:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:28:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:28:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:28:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:28:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:28:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:28:05' 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:28:05' 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:28:05' 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:28:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:28:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142084262mirror.0smithi16419.1.1-298-gb53b6d3efalseOK42104261mirror.1smithi16419.1.1-298-gb53b6d3efalseOK42164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":15938355.2,"bytes_per_snapshot":267946496.0,"last_snapshot_bytes":535892992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032479,"remote_snapshot_timestamp":1726032479,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:28:05cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:28:05' 2024-09-11T05:28:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:28:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:28:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:28:05' 2024-09-11T05:28:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:28:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:28:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:28:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:28:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap2 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2024-09-11T05:28:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:28:27.515 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:28:27.515 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:28:27.515 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:28:27.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:28:27.516 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:28:27.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:28:27.519 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-11T05:28:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:28:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:28:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:28:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:28:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:28:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:28:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:28:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:28:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:28:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:28:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:28:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:28:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:28:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:28:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:28:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:28:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:28:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:28:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:28:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:28:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:28:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:28:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:28:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:28:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:28:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:28:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:28:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:28:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:28:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:28:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:28:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:28:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:28:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:28:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:28:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:28:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:28:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:28:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:28:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:28:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:28:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:28:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:28:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:28:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:28:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:28:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:28:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:28:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:28:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:28:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:28:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:28:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:28:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:28:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:28:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:28:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:28:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:28:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:28:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:28:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 8 2024-09-11T05:28:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:28:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:28:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:28:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:28:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:28:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=8 2024-09-11T05:28:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:28:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:28:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:28:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:28:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:28:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 8 = '' 2024-09-11T05:28:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:28:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:28:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:28:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:28:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:28:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 8 = '' 2024-09-11T05:28:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:28:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:28:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:28:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:28:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:28:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 8 = '' 2024-09-11T05:28:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:28:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:28:57.506 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T05:28:57.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T05:28:57.507 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:28:57.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:28:57.505+0000 7f66f7bae640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 28912) UID: 1000 2024-09-11T05:28:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:28:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:28:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:28:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:28:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=8 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 8 = 8 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:28:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:28:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-11T05:28:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:28:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:28:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:28:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 4 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:28:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:28:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:28:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:28:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:28:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:28:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:28:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap1-clean ']' 2024-09-11T05:28:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T05:28:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:28:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:28:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:00.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:00.508 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T05:29:00.508 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:29:00.509 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T05:29:00.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T05:29:00.509 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T05:29:00.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T05:29:00.512 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-11T05:29:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap2-clean 2024-09-11T05:29:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-11T05:29:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2024-09-11T05:29:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:29:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:29:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:29:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:29:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:29:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:29:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:29:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 37235712 bytes 2024-09-11T05:29:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 55877632 bytes 2024-09-11T05:29:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 75310080 bytes 2024-09-11T05:29:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 107902976 bytes 2024-09-11T05:29:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 117755904 bytes 2024-09-11T05:29:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 135794688 bytes 2024-09-11T05:29:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 166650880 bytes 2024-09-11T05:29:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 167294976 bytes 2024-09-11T05:29:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 226639872 bytes 2024-09-11T05:29:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 269650944 bytes 2024-09-11T05:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 324721664 bytes 2024-09-11T05:29:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 363561984 bytes 2024-09-11T05:29:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 378710016 bytes 2024-09-11T05:29:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 444596224 bytes 2024-09-11T05:29:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 503457792 bytes 2024-09-11T05:29:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 535913472 bytes 2024-09-11T05:29:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 562301952 bytes 2024-09-11T05:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 622098432 bytes 2024-09-11T05:29:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 661871616 bytes 2024-09-11T05:29:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 663093248 bytes 2024-09-11T05:29:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 692759552 bytes 2024-09-11T05:29:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 715619328 bytes 2024-09-11T05:29:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 764475392 bytes 2024-09-11T05:29:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 814237696 bytes 2024-09-11T05:29:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 822929408 bytes 2024-09-11T05:29:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 843003904 bytes 2024-09-11T05:29:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 866664448 bytes 2024-09-11T05:29:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 890718208 bytes 2024-09-11T05:29:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 908521472 bytes 2024-09-11T05:29:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 945157120 bytes 2024-09-11T05:29:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1007946752 bytes 2024-09-11T05:29:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1035435008 bytes 2024-09-11T05:29:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1068644352 bytes 2024-09-11T05:29:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1117042688 bytes 2024-09-11T05:29:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1167361024 bytes 2024-09-11T05:29:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1207038976 bytes 2024-09-11T05:29:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1270284288 bytes 2024-09-11T05:29:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1297137664 bytes 2024-09-11T05:29:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1335962624 bytes 2024-09-11T05:29:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1396182016 bytes 2024-09-11T05:29:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1416727552 bytes 2024-09-11T05:29:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1544258560 bytes 2024-09-11T05:29:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1558812672 bytes 2024-09-11T05:29:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1657917440 bytes 2024-09-11T05:29:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1710790656 bytes 2024-09-11T05:29:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1735715840 bytes 2024-09-11T05:29:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1765336064 bytes 2024-09-11T05:29:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1800505344 bytes 2024-09-11T05:29:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1808914432 bytes 2024-09-11T05:29:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1849313280 bytes 2024-09-11T05:29:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1896667136 bytes 2024-09-11T05:29:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1913312256 bytes 2024-09-11T05:29:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1997847552 bytes 2024-09-11T05:29:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2079336448 bytes 2024-09-11T05:29:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2103438336 bytes 2024-09-11T05:29:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2156564480 bytes 2024-09-11T05:29:10.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2197400576 bytes 2024-09-11T05:29:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2234771456 bytes 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:29:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:29:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:29:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:29:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:29:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:29:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:29:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:29:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:29:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"} 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:29:05 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:29:05' 2024-09-11T05:29:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:29:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:29:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:29:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"} 2024-09-11T05:29:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:29:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:29:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:29:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:29:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:29:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:29:05' 2024-09-11T05:29:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:29:05' 2024-09-11T05:29:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:29:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:29:05' 2024-09-11T05:29:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:29:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:29:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:29:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142104261mirror.1smithi16419.1.1-298-gb53b6d3efalseOK42164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:29:05cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:29:05' 2024-09-11T05:29:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:29:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:29:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:29:05' 2024-09-11T05:29:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:29:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:29:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:29:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:29:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap3 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2024-09-11T05:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:29:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:29:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:29:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:29:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:29:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:29:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:29:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:29:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:29:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:29:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:29:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:29:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:29:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:29:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:29:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:29:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:29:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:29:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:29:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:29:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:29:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:29:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:29:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:29:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:29:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:29:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:29:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:29:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:29:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:29:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:29:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:29:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:29:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:29:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:29:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:29:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:29:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:29:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:29:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:29:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:29:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:29:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:29:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:29:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:29:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:29:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:29:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:29:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:29:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:29:44.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 11 2024-09-11T05:29:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:29:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:29:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:29:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:29:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=11 2024-09-11T05:29:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:29:44.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:29:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:29:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:29:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:29:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:29:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:29:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:29:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:29:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:29:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:29:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 11 = '' 2024-09-11T05:29:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:29:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:29:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:29:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:29:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:29:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:29:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:29:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:29:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:29:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:29:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 11 = '' 2024-09-11T05:29:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:29:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:29:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:29:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:29:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:29:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:29:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:29:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:29:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:29:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:29:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 11 = '' 2024-09-11T05:29:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:29:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:29:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:29:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:29:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:29:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=11 2024-09-11T05:29:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 11 = 11 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:29:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 6 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:29:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:29:00 2024 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:29:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:29:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap2-clean ']' 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T05:29:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2024-09-11T05:29:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:29:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:29:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:29:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap3-clean 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2024-09-11T05:29:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:29:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:29:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 31307776 bytes 2024-09-11T05:29:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 66810880 bytes 2024-09-11T05:29:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 67401728 bytes 2024-09-11T05:29:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 144218112 bytes 2024-09-11T05:29:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 222290944 bytes 2024-09-11T05:29:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 252478464 bytes 2024-09-11T05:29:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 313840640 bytes 2024-09-11T05:29:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 344506368 bytes 2024-09-11T05:29:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 375947264 bytes 2024-09-11T05:29:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 379925504 bytes 2024-09-11T05:29:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 389353472 bytes 2024-09-11T05:29:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 444801024 bytes 2024-09-11T05:29:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 476350464 bytes 2024-09-11T05:29:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 476976128 bytes 2024-09-11T05:29:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 522109952 bytes 2024-09-11T05:29:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 536570880 bytes 2024-09-11T05:29:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 550125568 bytes 2024-09-11T05:29:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 609812480 bytes 2024-09-11T05:29:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 643221504 bytes 2024-09-11T05:29:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 717115392 bytes 2024-09-11T05:29:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 793023488 bytes 2024-09-11T05:29:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 811455488 bytes 2024-09-11T05:29:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 812096512 bytes 2024-09-11T05:29:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 837378048 bytes 2024-09-11T05:29:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 878260224 bytes 2024-09-11T05:29:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 885969920 bytes 2024-09-11T05:29:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 908822528 bytes 2024-09-11T05:29:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 972634112 bytes 2024-09-11T05:29:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1004184576 bytes 2024-09-11T05:29:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1070434304 bytes 2024-09-11T05:29:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1098978304 bytes 2024-09-11T05:29:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1134387200 bytes 2024-09-11T05:29:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1182343168 bytes 2024-09-11T05:29:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1216453632 bytes 2024-09-11T05:29:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1264892928 bytes 2024-09-11T05:29:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1310998528 bytes 2024-09-11T05:29:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1359904768 bytes 2024-09-11T05:29:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1413436416 bytes 2024-09-11T05:29:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1459086336 bytes 2024-09-11T05:29:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1477680128 bytes 2024-09-11T05:29:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1527933952 bytes 2024-09-11T05:29:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1535745024 bytes 2024-09-11T05:29:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1553926144 bytes 2024-09-11T05:29:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1566177280 bytes 2024-09-11T05:29:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1606580224 bytes 2024-09-11T05:29:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1639843840 bytes 2024-09-11T05:29:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1678385152 bytes 2024-09-11T05:29:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1744752640 bytes 2024-09-11T05:29:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1774382080 bytes 2024-09-11T05:29:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1780658176 bytes 2024-09-11T05:30:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1881445376 bytes 2024-09-11T05:30:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1910577152 bytes 2024-09-11T05:30:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1934709760 bytes 2024-09-11T05:30:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1954529280 bytes 2024-09-11T05:30:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1983463424 bytes 2024-09-11T05:30:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 1985903616 bytes 2024-09-11T05:30:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2026118144 bytes 2024-09-11T05:30:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2061080576 bytes 2024-09-11T05:30:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2114743296 bytes 2024-09-11T05:30:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2155688960 bytes 2024-09-11T05:30:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2221992960 bytes 2024-09-11T05:30:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2287259648 bytes 2024-09-11T05:30:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2299241472 bytes 2024-09-11T05:30:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2337564672 bytes 2024-09-11T05:30:01.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2363390976 bytes 2024-09-11T05:30:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2438329344 bytes 2024-09-11T05:30:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2463079424 bytes 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:30:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:30:02.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2819835648075639) > weight (0.25) 2024-09-11T05:30:02.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4769433685694484) > weight (0.25) 2024-09-11T05:30:02.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3123442940808534) > weight (0.25) 2024-09-11T05:30:02.521 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T05:30:02.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T05:30:02.522 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T05:30:02.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:30:02.520+0000 7f1920861640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 28972) UID: 1000 2024-09-11T05:30:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:30:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:30:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:30:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:30:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:30:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"} 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:29:35 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+stopped 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:29:35' 2024-09-11T05:30:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:30:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:30:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:30:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"} 2024-09-11T05:30:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:30:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:30:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:30:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:30:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:30:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:30:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:30:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:30:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:29:35' 2024-09-11T05:30:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:29:35' 2024-09-11T05:30:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:29:35' 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:30:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:30:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='WARNINGOKWARNING142104261mirror.1smithi16419.1.1-298-gb53b6d3efalseOK42164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17888699.73,"bytes_per_snapshot":536276992.0,"last_snapshot_bytes":536660992,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032534,"remote_snapshot_timestamp":1726032534,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:29:35cluster275c5b592-5745-4d1a-984f-ff4a3feb0219down+stoppedlocal image is primary2024-09-11 05:29:35' 2024-09-11T05:30:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:30:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:30:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:29:35' 2024-09-11T05:30:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:30:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:30:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:30:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:30:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:30:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap4 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2024-09-11T05:30:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:30:10.522 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T05:30:10.523 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:30:10.523 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T05:30:10.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:30:10.523 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:30:10.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:30:10.526 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-11T05:30:20.512 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5137652553394507) > weight (0.25) 2024-09-11T05:30:20.513 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-11T05:30:20.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-11T05:30:20.514 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T05:30:20.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T05:30:20.512+0000 7fe2421e0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 28914) UID: 1000 2024-09-11T05:30:22.514 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-11T05:30:22.514 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:30:22.515 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-11T05:30:22.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T05:30:22.515 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T05:30:22.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T05:30:22.518 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:30:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:30:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:30:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:30:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:30:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:30:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:30:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:30:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:30:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:30:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:30:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:30:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:30:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:30:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:30:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:30:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:30:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:30:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:30:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:30:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:30:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:30:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:30:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:30:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:30:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:30:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:30:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:30:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:30:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:30:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:30:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:30:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:30:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:30:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:30:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:30:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:30:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:30:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:30:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:30:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:30:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:30:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:30:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:30:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:30:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:30:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:30:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:30:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:30:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:30:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:30:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:30:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:30:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:30:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:30:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:30:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:30:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:30:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:30:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:30:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:30:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 14 2024-09-11T05:30:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:30:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:30:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:30:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=14 2024-09-11T05:30:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:30:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:30:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:30:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:30:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:30:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:30:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:30:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:30:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:30:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:30:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:30:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 14 = '' 2024-09-11T05:30:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:30:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:30:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:30:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:30:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:30:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:30:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:30:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:30:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:30:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:30:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 14 = '' 2024-09-11T05:30:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:30:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:30:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:30:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:30:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:30:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:30:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:30:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:30:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:30:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:30:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 14 = '' 2024-09-11T05:30:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:30:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:30:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:30:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:30:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=14 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 14 = 14 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2024-09-11T05:30:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:30:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:30:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:30:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 8 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:30:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:29:00 2024 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:30:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap3-clean ']' 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:30:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:30:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:30:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2024-09-11T05:30:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:30:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:30:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:30:42.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:30:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:30:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:30:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap4-clean 2024-09-11T05:30:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-11T05:30:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2024-09-11T05:30:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:30:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:30:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:30:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:30:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 58816512 bytes 2024-09-11T05:30:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 200732672 bytes 2024-09-11T05:30:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 263368704 bytes 2024-09-11T05:30:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 284344320 bytes 2024-09-11T05:30:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 340840448 bytes 2024-09-11T05:30:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 374517760 bytes 2024-09-11T05:30:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 392175616 bytes 2024-09-11T05:30:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 440825856 bytes 2024-09-11T05:30:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 500308992 bytes 2024-09-11T05:30:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 524990464 bytes 2024-09-11T05:30:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 536632320 bytes 2024-09-11T05:30:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 571745280 bytes 2024-09-11T05:30:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 572336128 bytes 2024-09-11T05:30:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 656206848 bytes 2024-09-11T05:30:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 717429760 bytes 2024-09-11T05:30:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 747284480 bytes 2024-09-11T05:30:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 820721664 bytes 2024-09-11T05:30:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 840324096 bytes 2024-09-11T05:30:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 866837504 bytes 2024-09-11T05:30:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 969630720 bytes 2024-09-11T05:30:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 977300480 bytes 2024-09-11T05:30:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 977916928 bytes 2024-09-11T05:30:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1008285696 bytes 2024-09-11T05:30:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1053911040 bytes 2024-09-11T05:30:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1096819712 bytes 2024-09-11T05:30:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1141322752 bytes 2024-09-11T05:30:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1195555840 bytes 2024-09-11T05:30:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1294940160 bytes 2024-09-11T05:30:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1316071424 bytes 2024-09-11T05:30:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1354571776 bytes 2024-09-11T05:30:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1355189248 bytes 2024-09-11T05:30:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1392696320 bytes 2024-09-11T05:30:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1457308672 bytes 2024-09-11T05:30:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1508627456 bytes 2024-09-11T05:30:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1568254976 bytes 2024-09-11T05:30:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1613735936 bytes 2024-09-11T05:30:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1624849408 bytes 2024-09-11T05:30:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1644633088 bytes 2024-09-11T05:30:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1675196416 bytes 2024-09-11T05:30:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1716785152 bytes 2024-09-11T05:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1732132864 bytes 2024-09-11T05:30:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1778839552 bytes 2024-09-11T05:30:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1808411648 bytes 2024-09-11T05:30:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1845615616 bytes 2024-09-11T05:30:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1902924800 bytes 2024-09-11T05:30:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1924645888 bytes 2024-09-11T05:30:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1992249344 bytes 2024-09-11T05:30:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2018398208 bytes 2024-09-11T05:30:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2064940032 bytes 2024-09-11T05:30:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2103565312 bytes 2024-09-11T05:30:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2149506048 bytes 2024-09-11T05:30:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2185125888 bytes 2024-09-11T05:30:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2216839168 bytes 2024-09-11T05:30:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2288828416 bytes 2024-09-11T05:30:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2295227392 bytes 2024-09-11T05:30:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2321910784 bytes 2024-09-11T05:30:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2322982912 bytes 2024-09-11T05:30:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2389557248 bytes 2024-09-11T05:30:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2414691328 bytes 2024-09-11T05:30:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2451419136 bytes 2024-09-11T05:30:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2455501824 bytes 2024-09-11T05:30:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2537474048 bytes 2024-09-11T05:30:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2611555328 bytes 2024-09-11T05:30:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2652900352 bytes 2024-09-11T05:30:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2694558720 bytes 2024-09-11T05:30:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2715203584 bytes 2024-09-11T05:30:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2720040960 bytes 2024-09-11T05:30:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2748746752 bytes 2024-09-11T05:30:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2785810432 bytes 2024-09-11T05:30:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2828717056 bytes 2024-09-11T05:30:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2829344768 bytes 2024-09-11T05:30:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2889345024 bytes 2024-09-11T05:30:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2936105984 bytes 2024-09-11T05:30:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3001427968 bytes 2024-09-11T05:30:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3057110016 bytes 2024-09-11T05:30:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3112216576 bytes 2024-09-11T05:30:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3147515904 bytes 2024-09-11T05:30:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3269058560 bytes 2024-09-11T05:30:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3310594048 bytes 2024-09-11T05:30:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3385297920 bytes 2024-09-11T05:30:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3422707712 bytes 2024-09-11T05:30:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3470867456 bytes 2024-09-11T05:30:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3504190464 bytes 2024-09-11T05:30:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3590326272 bytes 2024-09-11T05:30:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3623216128 bytes 2024-09-11T05:30:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3658326016 bytes 2024-09-11T05:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3695877120 bytes 2024-09-11T05:30:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3724660736 bytes 2024-09-11T05:30:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3728048128 bytes 2024-09-11T05:30:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3766585344 bytes 2024-09-11T05:30:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3797385216 bytes 2024-09-11T05:30:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3813026816 bytes 2024-09-11T05:30:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3847814144 bytes 2024-09-11T05:30:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3886759936 bytes 2024-09-11T05:30:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3933877248 bytes 2024-09-11T05:30:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3980186624 bytes 2024-09-11T05:30:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4010768384 bytes 2024-09-11T05:30:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4074299392 bytes 2024-09-11T05:30:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4105542656 bytes 2024-09-11T05:30:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4172713984 bytes 2024-09-11T05:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4207121408 bytes 2024-09-11T05:30:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4311455744 bytes 2024-09-11T05:30:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4336983040 bytes 2024-09-11T05:30:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4378865664 bytes 2024-09-11T05:30:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4395187200 bytes 2024-09-11T05:30:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4424840192 bytes 2024-09-11T05:30:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4472690688 bytes 2024-09-11T05:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:30:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:30:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:30:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:30:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:30:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:30:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531475456.0,"last_snapshot_bytes":526289920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032583,"remote_snapshot_timestamp":1726032583,"replay_state":"idle"} 2024-09-11T05:30:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:30:35 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:30:40' 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:30:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:30:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:30:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531475456.0,"last_snapshot_bytes":526289920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032583,"remote_snapshot_timestamp":1726032583,"replay_state":"idle"} 2024-09-11T05:30:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:30:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:30:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:30:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:30:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:30:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:30:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:30:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:30:35' 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:30:35' 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:30:35' 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:30:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:30:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531475456.0,"last_snapshot_bytes":526289920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032583,"remote_snapshot_timestamp":1726032583,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:30:35cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:30:40' 2024-09-11T05:30:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:30:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:30:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:30:35' 2024-09-11T05:30:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:30:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap5 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2024-09-11T05:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:31:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:31:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:31:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:31:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:31:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:31:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:31:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:31:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:31:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:31:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:31:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:31:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:31:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:31:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:31:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:31:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:31:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:31:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:31:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:31:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:31:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:31:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:31:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:31:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:31:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:31:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:31:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:31:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:31:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:31:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:31:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:31:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:31:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:31:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:31:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:31:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:31:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:31:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:31:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:31:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:31:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:31:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:31:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:31:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:31:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:31:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:31:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:31:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:31:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:31:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:31:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:31:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:31:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:31:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:31:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:31:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:31:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:31:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 17 2024-09-11T05:31:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:31:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:31:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:31:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:31:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:31:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=17 2024-09-11T05:31:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:31:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:31:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:31:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:31:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:31:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:31:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:31:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:31:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:31:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = '' 2024-09-11T05:31:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:31:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:31:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:31:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:31:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:31:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:31:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:31:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:31:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:31:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:31:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = '' 2024-09-11T05:31:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:31:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:31:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:31:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:31:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:31:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:31:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:31:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:31:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:31:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = '' 2024-09-11T05:31:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:31:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:31:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:31:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:31:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:31:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:31:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=17 2024-09-11T05:31:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = 17 2024-09-11T05:31:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:31:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-11T05:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:31:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 10 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:30:40 2024 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:31:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:31:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:31:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap4-clean ']' 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:31:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:31:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:31:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:31:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:31:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:31:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:31:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:31:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:31:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap5-clean 2024-09-11T05:31:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-11T05:31:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2024-09-11T05:31:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:31:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:31:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:31:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:31:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:31:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:31:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:31:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 33138688 bytes 2024-09-11T05:31:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 47321088 bytes 2024-09-11T05:31:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 118923264 bytes 2024-09-11T05:31:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 173608960 bytes 2024-09-11T05:31:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 232408064 bytes 2024-09-11T05:31:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 284054528 bytes 2024-09-11T05:31:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 318807040 bytes 2024-09-11T05:31:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 343071744 bytes 2024-09-11T05:31:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 358581248 bytes 2024-09-11T05:31:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 373746688 bytes 2024-09-11T05:31:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 426569728 bytes 2024-09-11T05:31:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 462727168 bytes 2024-09-11T05:31:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 492110848 bytes 2024-09-11T05:31:41.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 557665280 bytes 2024-09-11T05:31:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 576698368 bytes 2024-09-11T05:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 594032640 bytes 2024-09-11T05:31:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 638459904 bytes 2024-09-11T05:31:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 686909440 bytes 2024-09-11T05:31:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 711786496 bytes 2024-09-11T05:31:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 755129344 bytes 2024-09-11T05:31:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 759934976 bytes 2024-09-11T05:31:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 784232448 bytes 2024-09-11T05:31:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 826194944 bytes 2024-09-11T05:31:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 828802048 bytes 2024-09-11T05:31:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 907359232 bytes 2024-09-11T05:31:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 922883072 bytes 2024-09-11T05:31:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 992624640 bytes 2024-09-11T05:31:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1032155136 bytes 2024-09-11T05:31:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1045079040 bytes 2024-09-11T05:31:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1068572672 bytes 2024-09-11T05:31:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1121597440 bytes 2024-09-11T05:31:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1122209792 bytes 2024-09-11T05:31:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1141425152 bytes 2024-09-11T05:31:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1201506304 bytes 2024-09-11T05:31:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1240620032 bytes 2024-09-11T05:31:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1260244992 bytes 2024-09-11T05:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1285626880 bytes 2024-09-11T05:31:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1354284032 bytes 2024-09-11T05:31:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1437877248 bytes 2024-09-11T05:31:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1464140800 bytes 2024-09-11T05:31:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1511309312 bytes 2024-09-11T05:31:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1580705792 bytes 2024-09-11T05:31:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1671514112 bytes 2024-09-11T05:31:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1691292672 bytes 2024-09-11T05:31:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1692810240 bytes 2024-09-11T05:31:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1730735104 bytes 2024-09-11T05:31:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1755936768 bytes 2024-09-11T05:31:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1757025280 bytes 2024-09-11T05:31:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1779294208 bytes 2024-09-11T05:31:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1780294656 bytes 2024-09-11T05:31:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1886228480 bytes 2024-09-11T05:31:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1888113664 bytes 2024-09-11T05:31:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1914963968 bytes 2024-09-11T05:31:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1962383360 bytes 2024-09-11T05:31:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1994561536 bytes 2024-09-11T05:31:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2065838080 bytes 2024-09-11T05:31:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2143422464 bytes 2024-09-11T05:31:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2149052416 bytes 2024-09-11T05:31:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2176662528 bytes 2024-09-11T05:31:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2231269376 bytes 2024-09-11T05:31:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2288360448 bytes 2024-09-11T05:31:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2315655168 bytes 2024-09-11T05:31:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2375573504 bytes 2024-09-11T05:31:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2403158016 bytes 2024-09-11T05:31:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2468552704 bytes 2024-09-11T05:31:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2499061760 bytes 2024-09-11T05:31:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2564097024 bytes 2024-09-11T05:31:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2588915712 bytes 2024-09-11T05:31:47.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4959770653410407) > weight (0.25) 2024-09-11T05:31:47.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.782203026291461) > weight (0.25) 2024-09-11T05:31:47.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.29245747518374754) > weight (0.25) 2024-09-11T05:31:47.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5590884998711787) > weight (0.25) 2024-09-11T05:31:47.519 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-11T05:31:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2659641344 bytes 2024-09-11T05:31:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2698698752 bytes 2024-09-11T05:31:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2787190784 bytes 2024-09-11T05:31:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2850988032 bytes 2024-09-11T05:31:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2880615424 bytes 2024-09-11T05:31:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2925287424 bytes 2024-09-11T05:31:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2972798976 bytes 2024-09-11T05:31:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3055296512 bytes 2024-09-11T05:31:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3086960640 bytes 2024-09-11T05:31:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3120572416 bytes 2024-09-11T05:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3187570688 bytes 2024-09-11T05:31:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3252591616 bytes 2024-09-11T05:31:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3292959744 bytes 2024-09-11T05:31:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3320873984 bytes 2024-09-11T05:31:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3378702336 bytes 2024-09-11T05:31:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3404285952 bytes 2024-09-11T05:31:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3444916224 bytes 2024-09-11T05:31:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3520665600 bytes 2024-09-11T05:31:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3528793088 bytes 2024-09-11T05:31:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3595901952 bytes 2024-09-11T05:31:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3605280768 bytes 2024-09-11T05:31:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3661264896 bytes 2024-09-11T05:31:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3690798080 bytes 2024-09-11T05:31:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3736070144 bytes 2024-09-11T05:31:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3764969472 bytes 2024-09-11T05:31:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3866752000 bytes 2024-09-11T05:31:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3929084928 bytes 2024-09-11T05:31:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3984568320 bytes 2024-09-11T05:31:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 3994727424 bytes 2024-09-11T05:31:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4029926400 bytes 2024-09-11T05:31:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4101130240 bytes 2024-09-11T05:31:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4136171520 bytes 2024-09-11T05:31:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4189386752 bytes 2024-09-11T05:31:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4250648576 bytes 2024-09-11T05:31:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4292540416 bytes 2024-09-11T05:31:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4299387904 bytes 2024-09-11T05:31:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4317395968 bytes 2024-09-11T05:31:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4338416640 bytes 2024-09-11T05:31:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4374355968 bytes 2024-09-11T05:31:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10800 IOs, 4399691776 bytes 2024-09-11T05:31:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10900 IOs, 4451903488 bytes 2024-09-11T05:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11000 IOs, 4452506624 bytes 2024-09-11T05:31:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11100 IOs, 4520384512 bytes 2024-09-11T05:31:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11200 IOs, 4521015296 bytes 2024-09-11T05:31:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11300 IOs, 4551693312 bytes 2024-09-11T05:31:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11400 IOs, 4617468928 bytes 2024-09-11T05:31:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11500 IOs, 4686429184 bytes 2024-09-11T05:31:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11600 IOs, 4712484864 bytes 2024-09-11T05:31:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11700 IOs, 4761465856 bytes 2024-09-11T05:31:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11800 IOs, 4783901696 bytes 2024-09-11T05:31:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11900 IOs, 4850024448 bytes 2024-09-11T05:31:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12000 IOs, 4857172992 bytes 2024-09-11T05:31:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12100 IOs, 4890378240 bytes 2024-09-11T05:31:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12200 IOs, 4935176192 bytes 2024-09-11T05:31:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:12300 IOs, 4961309696 bytes 2024-09-11T05:31:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:31:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:31:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:31:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:31:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:31:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:31:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:31:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:31:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:31:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:31:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17813230.93,"bytes_per_snapshot":534736384.0,"last_snapshot_bytes":534396928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032689,"remote_snapshot_timestamp":1726032689,"replay_state":"idle"} 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:31:35 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:31:40' 2024-09-11T05:31:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:31:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:31:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:31:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17813230.93,"bytes_per_snapshot":534736384.0,"last_snapshot_bytes":534396928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032689,"remote_snapshot_timestamp":1726032689,"replay_state":"idle"} 2024-09-11T05:31:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:31:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:31:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:31:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:31:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:31:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:31:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:31:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:31:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:31:35' 2024-09-11T05:31:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:31:35' 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:31:35' 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:31:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17813230.93,"bytes_per_snapshot":534736384.0,"last_snapshot_bytes":534396928,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726032689,"remote_snapshot_timestamp":1726032689,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:31:35cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:31:40' 2024-09-11T05:31:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:31:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:31:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:31:35' 2024-09-11T05:31:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:31:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:31:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:31:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:31:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap6 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2024-09-11T05:31:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:31:55.528 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:31:55.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:31:55.528 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T05:31:55.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:31:55.527+0000 7fe13064c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 31235) UID: 1000 2024-09-11T05:31:57.529 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:31:57.529 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:31:57.530 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:31:57.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:31:57.530 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:31:57.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:31:57.533 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-11T05:32:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:32:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:32:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:32:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:32:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:32:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:32:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:32:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:32:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:32:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:32:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:32:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:32:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:32:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:32:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:32:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:32:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:32:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:32:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:32:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:32:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:32:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:32:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:32:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:32:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:32:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:32:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:32:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:32:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:32:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:32:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:32:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:32:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:32:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:32:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:32:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:32:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:32:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:32:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:32:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:2 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:2 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:32:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=2 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=2 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:2 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:2 2024-09-11T05:32:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=2 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=2 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=2 2024-09-11T05:32:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T05:32:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:32:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:32:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T05:32:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T05:32:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:32:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:32:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:32:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:32:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:32:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:32:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:32:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:32:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:32:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:32:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:32:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 20 2024-09-11T05:32:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:32:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:32:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:32:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:32:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:32:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=20 2024-09-11T05:32:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:32:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:32:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:32:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:32:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:32:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:32:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:32:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:32:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:32:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:32:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:32:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 20 = '' 2024-09-11T05:32:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:32:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:32:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:32:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:32:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:32:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:32:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 20 = '' 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:32:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:32:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:32:29.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:32:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:32:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:32:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:32:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:32:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:32:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 20 = '' 2024-09-11T05:32:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:32:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:32:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:32:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:32:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:32:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=20 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 20 = 20 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:32:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:32:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:32:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-11T05:32:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:32:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:32:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 12 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:30:40 2024 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:32:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:32:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:32:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:32:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:32:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap5-clean ']' 2024-09-11T05:32:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T05:32:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:32:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:32:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:32:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T05:32:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:32:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:32:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:32:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:32:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap6-clean 2024-09-11T05:32:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-11T05:32:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2024-09-11T05:32:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:32:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:32:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:32:36.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:32:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 47327232 bytes 2024-09-11T05:32:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 98930688 bytes 2024-09-11T05:32:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 111895552 bytes 2024-09-11T05:32:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 123787264 bytes 2024-09-11T05:32:37.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 182087680 bytes 2024-09-11T05:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 232376320 bytes 2024-09-11T05:32:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 267800576 bytes 2024-09-11T05:32:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 306557952 bytes 2024-09-11T05:32:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 322636800 bytes 2024-09-11T05:32:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 360139776 bytes 2024-09-11T05:32:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 380283904 bytes 2024-09-11T05:32:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 427555840 bytes 2024-09-11T05:32:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 469448704 bytes 2024-09-11T05:32:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 517036032 bytes 2024-09-11T05:32:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 597412864 bytes 2024-09-11T05:32:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 623460352 bytes 2024-09-11T05:32:38.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 660327424 bytes 2024-09-11T05:32:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 694972416 bytes 2024-09-11T05:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 734941184 bytes 2024-09-11T05:32:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 811491328 bytes 2024-09-11T05:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 871774208 bytes 2024-09-11T05:32:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 882227200 bytes 2024-09-11T05:32:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 882828288 bytes 2024-09-11T05:32:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 900969472 bytes 2024-09-11T05:32:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 995437568 bytes 2024-09-11T05:32:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1062900736 bytes 2024-09-11T05:32:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1133218816 bytes 2024-09-11T05:32:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1192475648 bytes 2024-09-11T05:32:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1240981504 bytes 2024-09-11T05:32:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1273100288 bytes 2024-09-11T05:32:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1363857408 bytes 2024-09-11T05:32:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1418478592 bytes 2024-09-11T05:32:40.876 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:32:40.870+0000 7f14da894640 -1 reset not still connected to 0x556432daa680 2024-09-11T05:32:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1488984064 bytes 2024-09-11T05:32:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1502104576 bytes 2024-09-11T05:32:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1513833472 bytes 2024-09-11T05:32:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1518998528 bytes 2024-09-11T05:32:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1580376064 bytes 2024-09-11T05:32:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1680489472 bytes 2024-09-11T05:32:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1688046592 bytes 2024-09-11T05:32:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1767863296 bytes 2024-09-11T05:32:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1867139072 bytes 2024-09-11T05:32:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1919515648 bytes 2024-09-11T05:32:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1937818624 bytes 2024-09-11T05:32:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1958530048 bytes 2024-09-11T05:32:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2043876352 bytes 2024-09-11T05:32:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2054059008 bytes 2024-09-11T05:32:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2111282176 bytes 2024-09-11T05:32:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2123972608 bytes 2024-09-11T05:32:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2141620224 bytes 2024-09-11T05:32:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2165159936 bytes 2024-09-11T05:32:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2190474240 bytes 2024-09-11T05:32:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2231948288 bytes 2024-09-11T05:32:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2270139392 bytes 2024-09-11T05:32:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2364925952 bytes 2024-09-11T05:32:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2387908608 bytes 2024-09-11T05:32:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2413010944 bytes 2024-09-11T05:32:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:32:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:32:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:32:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:32:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:32:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:32:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17842619.73,"bytes_per_snapshot":534837760.0,"last_snapshot_bytes":535278592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032746,"remote_snapshot_timestamp":1726032746,"replay_state":"idle"} 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:32:35 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:32:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:32:40' 2024-09-11T05:32:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:32:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:32:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:32:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17842619.73,"bytes_per_snapshot":534837760.0,"last_snapshot_bytes":535278592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032746,"remote_snapshot_timestamp":1726032746,"replay_state":"idle"} 2024-09-11T05:32:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:32:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:32:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:32:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:32:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:32:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2024-09-11T05:32:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:32:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:32:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:32:35' 2024-09-11T05:32:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:32:35' 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:32:35' 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:32:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:32:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142164265mirror.2smithi16419.1.1-298-gb53b6d3etrueOK42184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17842619.73,"bytes_per_snapshot":534837760.0,"last_snapshot_bytes":535278592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032746,"remote_snapshot_timestamp":1726032746,"replay_state":"idle"}42164265mirror.2smithi1642024-09-11 05:32:35cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:32:40' 2024-09-11T05:32:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:32:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:32:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:32:35' 2024-09-11T05:32:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:32:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:32:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:32:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:32:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap7 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2024-09-11T05:32:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:32:48.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5922179101616192) > weight (0.25) 2024-09-11T05:32:48.521 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6814386456982493) > weight (0.25) 2024-09-11T05:32:48.521 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:32:48.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:32:48.521 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T05:32:48.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:32:48.518+0000 7f057677e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 28916) UID: 1000 2024-09-11T05:32:52.522 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:32:52.522 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:32:52.523 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:32:52.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:32:52.523 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:32:52.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:32:52.526 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-11T05:33:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:33:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:33:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:33:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:33:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:33:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:33:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:33:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:33:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:33:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:33:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:33:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:33:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:33:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:33:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:33:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:33:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:33:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:33:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:33:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:33:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:33:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:33:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:33:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:33:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:33:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:33:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:33:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:33:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:33:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:33:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:33:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:33:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:33:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:33:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:33:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:33:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:33:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:33:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 23 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:33:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:33:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:33:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:33:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=23 2024-09-11T05:33:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:33:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:33:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:33:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:33:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:33:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:33:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:33:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:33:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:33:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 23 = '' 2024-09-11T05:33:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:33:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:33:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:33:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:33:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:33:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:33:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:33:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:33:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:33:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:33:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 23 = '' 2024-09-11T05:33:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:33:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:33:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:33:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:33:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:33:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:33:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:33:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:33:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 23 = '' 2024-09-11T05:33:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:33:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:33:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:33:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:33:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:33:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:33:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=23 2024-09-11T05:33:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 23 = 23 2024-09-11T05:33:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-11T05:33:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:33:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:33:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 14 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:33:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:32:33 2024 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap6-clean ']' 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:33:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:33:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:33:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:33:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T05:33:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:33:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:33:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:33:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap7-clean 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2024-09-11T05:33:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:33:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:33:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:33:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:33:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:33:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:33:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:33:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 22053888 bytes 2024-09-11T05:33:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 73772032 bytes 2024-09-11T05:33:28.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 103713792 bytes 2024-09-11T05:33:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 152737792 bytes 2024-09-11T05:33:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 245349376 bytes 2024-09-11T05:33:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 296127488 bytes 2024-09-11T05:33:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 396100608 bytes 2024-09-11T05:33:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 445486080 bytes 2024-09-11T05:33:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 456699904 bytes 2024-09-11T05:33:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 499346432 bytes 2024-09-11T05:33:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 539667456 bytes 2024-09-11T05:33:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 555584512 bytes 2024-09-11T05:33:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 588696576 bytes 2024-09-11T05:33:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 625872896 bytes 2024-09-11T05:33:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 668150784 bytes 2024-09-11T05:33:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 680401920 bytes 2024-09-11T05:33:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 688706560 bytes 2024-09-11T05:33:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 712563712 bytes 2024-09-11T05:33:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 743076864 bytes 2024-09-11T05:33:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 762457088 bytes 2024-09-11T05:33:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 768123904 bytes 2024-09-11T05:33:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 780488704 bytes 2024-09-11T05:33:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 874878976 bytes 2024-09-11T05:33:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 891011072 bytes 2024-09-11T05:33:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 942720000 bytes 2024-09-11T05:33:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 978898944 bytes 2024-09-11T05:33:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1065877504 bytes 2024-09-11T05:33:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1118499840 bytes 2024-09-11T05:33:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1138214912 bytes 2024-09-11T05:33:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1177489408 bytes 2024-09-11T05:33:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1241575424 bytes 2024-09-11T05:33:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1254564864 bytes 2024-09-11T05:33:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1320608768 bytes 2024-09-11T05:33:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1383664640 bytes 2024-09-11T05:33:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1426054144 bytes 2024-09-11T05:33:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1475772416 bytes 2024-09-11T05:33:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1522235392 bytes 2024-09-11T05:33:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1565054976 bytes 2024-09-11T05:33:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1607780352 bytes 2024-09-11T05:33:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1639385088 bytes 2024-09-11T05:33:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1658324992 bytes 2024-09-11T05:33:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1723264000 bytes 2024-09-11T05:33:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1754199040 bytes 2024-09-11T05:33:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1766256640 bytes 2024-09-11T05:33:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1798847488 bytes 2024-09-11T05:33:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1842305024 bytes 2024-09-11T05:33:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1902787584 bytes 2024-09-11T05:33:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1977171968 bytes 2024-09-11T05:33:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2038541312 bytes 2024-09-11T05:33:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2128284672 bytes 2024-09-11T05:33:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2146449408 bytes 2024-09-11T05:33:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2223217664 bytes 2024-09-11T05:33:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2285904896 bytes 2024-09-11T05:33:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2294943744 bytes 2024-09-11T05:33:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2324192256 bytes 2024-09-11T05:33:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2435183616 bytes 2024-09-11T05:33:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2479030272 bytes 2024-09-11T05:33:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2517530624 bytes 2024-09-11T05:33:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2518210560 bytes 2024-09-11T05:33:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2540145664 bytes 2024-09-11T05:33:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2553754624 bytes 2024-09-11T05:33:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2563250176 bytes 2024-09-11T05:33:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2598964224 bytes 2024-09-11T05:33:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2643170304 bytes 2024-09-11T05:33:35.534 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:33:35.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:33:35.535 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:33:35.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:33:35.534+0000 7fd13ddef640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 33173) UID: 1000 2024-09-11T05:33:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2656157696 bytes 2024-09-11T05:33:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2706919424 bytes 2024-09-11T05:33:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2722055168 bytes 2024-09-11T05:33:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2806550528 bytes 2024-09-11T05:33:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2850195456 bytes 2024-09-11T05:33:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2859572224 bytes 2024-09-11T05:33:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2924646400 bytes 2024-09-11T05:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2960131072 bytes 2024-09-11T05:33:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2992412672 bytes 2024-09-11T05:33:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3026168832 bytes 2024-09-11T05:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3033271296 bytes 2024-09-11T05:33:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3069106176 bytes 2024-09-11T05:33:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3074210816 bytes 2024-09-11T05:33:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3117852672 bytes 2024-09-11T05:33:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3154391040 bytes 2024-09-11T05:33:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3205317632 bytes 2024-09-11T05:33:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3298502656 bytes 2024-09-11T05:33:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3340906496 bytes 2024-09-11T05:33:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3382424576 bytes 2024-09-11T05:33:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3416850432 bytes 2024-09-11T05:33:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3449704448 bytes 2024-09-11T05:33:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3504490496 bytes 2024-09-11T05:33:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3505121280 bytes 2024-09-11T05:33:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3570104320 bytes 2024-09-11T05:33:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3593612288 bytes 2024-09-11T05:33:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3594245120 bytes 2024-09-11T05:33:38.536 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:33:38.536 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:33:38.537 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:33:38.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:33:38.537 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:33:38.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:33:38.541 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-11T05:33:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3594859520 bytes 2024-09-11T05:33:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3673139200 bytes 2024-09-11T05:33:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3673764864 bytes 2024-09-11T05:33:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3726759936 bytes 2024-09-11T05:33:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3750575104 bytes 2024-09-11T05:33:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3773499392 bytes 2024-09-11T05:33:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 3841455104 bytes 2024-09-11T05:33:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 3852411904 bytes 2024-09-11T05:33:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 3892300800 bytes 2024-09-11T05:33:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 3965658112 bytes 2024-09-11T05:33:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4012491776 bytes 2024-09-11T05:33:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4081218560 bytes 2024-09-11T05:33:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4169202688 bytes 2024-09-11T05:33:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4216259584 bytes 2024-09-11T05:33:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4239926272 bytes 2024-09-11T05:33:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4277154816 bytes 2024-09-11T05:33:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4344409088 bytes 2024-09-11T05:33:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:33:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:33:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:33:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:33:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:33:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:33:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:33:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:33:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:33:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:33:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:33:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17734929.07,"bytes_per_snapshot":532047872.0,"last_snapshot_bytes":532047872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032797,"remote_snapshot_timestamp":1726032797,"replay_state":"idle"} 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.1 on smithi164 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:33:22 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:33:40' 2024-09-11T05:33:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:33:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:33:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:33:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17734929.07,"bytes_per_snapshot":532047872.0,"last_snapshot_bytes":532047872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032797,"remote_snapshot_timestamp":1726032797,"replay_state":"idle"} 2024-09-11T05:33:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:33:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:33:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:33:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:33:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:33:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.1 on smithi164 2024-09-11T05:33:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:33:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:33:22' 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:33:22' 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:33:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:33:22' 2024-09-11T05:33:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:33:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:33:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:33:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3etrueOK45084511mirror.2smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17734929.07,"bytes_per_snapshot":532047872.0,"last_snapshot_bytes":532047872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032797,"remote_snapshot_timestamp":1726032797,"replay_state":"idle"}44114414mirror.1smithi1642024-09-11 05:33:22cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:33:40' 2024-09-11T05:33:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:33:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:33:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:33:22' 2024-09-11T05:33:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:33:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:33:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:33:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:33:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap8 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:33:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2024-09-11T05:33:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:34:07.526 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7170272933167425) > weight (0.25) 2024-09-11T05:34:07.527 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5374146237142134) > weight (0.25) 2024-09-11T05:34:07.527 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:34:07.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:34:07.527 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:34:07.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:34:07.529+0000 7f1661034640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 33653) UID: 1000 2024-09-11T05:34:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:34:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:34:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:34:10.529 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:34:10.529 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:34:10.529 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:34:10.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:34:10.530 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:34:10.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:34:10.532 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-11T05:34:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:34:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:34:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:34:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:34:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:34:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:34:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:34:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:34:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:34:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:34:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:34:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:34:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:34:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:34:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:34:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:34:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:34:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:34:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:34:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:34:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:34:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:34:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:34:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:34:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:34:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:34:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:34:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:34:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:34:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:34:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:34:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:34:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:34:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:34:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:34:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:34:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:34:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:34:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:34:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:34:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 26 2024-09-11T05:34:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:34:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:34:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:34:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:34:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:34:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=26 2024-09-11T05:34:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:34:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:34:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:34:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:34:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:34:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:34:13.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:34:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:34:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:34:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:34:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:34:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 26 = '' 2024-09-11T05:34:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:34:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:34:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:34:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:34:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:34:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:34:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 26 = '' 2024-09-11T05:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:34:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:34:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:34:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:34:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:34:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:34:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:34:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 26 = '' 2024-09-11T05:34:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:34:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:34:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:34:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:34:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:34:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=26 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 26 = 26 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2024-09-11T05:34:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:34:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:34:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:34:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 16 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:34:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:32:33 2024 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:34:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:34:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:34:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap7-clean ']' 2024-09-11T05:34:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:34:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:34:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:34:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:34:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:34:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:34:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap8-clean 2024-09-11T05:34:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-11T05:34:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2024-09-11T05:34:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:34:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:34:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:34:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:34:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2024-09-11T05:34:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:34:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:34:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 71346176 bytes 2024-09-11T05:34:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 130427904 bytes 2024-09-11T05:34:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 157834240 bytes 2024-09-11T05:34:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 169413632 bytes 2024-09-11T05:34:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 236996608 bytes 2024-09-11T05:34:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 315859968 bytes 2024-09-11T05:34:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 373238784 bytes 2024-09-11T05:34:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 450010112 bytes 2024-09-11T05:34:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 523570176 bytes 2024-09-11T05:34:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 566938624 bytes 2024-09-11T05:34:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 628465664 bytes 2024-09-11T05:34:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 698647552 bytes 2024-09-11T05:34:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 741294080 bytes 2024-09-11T05:34:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 825303040 bytes 2024-09-11T05:34:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 910217216 bytes 2024-09-11T05:34:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 917371904 bytes 2024-09-11T05:34:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 939344896 bytes 2024-09-11T05:34:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 988965888 bytes 2024-09-11T05:34:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1010696192 bytes 2024-09-11T05:34:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1043843072 bytes 2024-09-11T05:34:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1117404160 bytes 2024-09-11T05:34:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1157870592 bytes 2024-09-11T05:34:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1194854400 bytes 2024-09-11T05:34:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1234939904 bytes 2024-09-11T05:34:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1249809408 bytes 2024-09-11T05:34:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1291180032 bytes 2024-09-11T05:34:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1360314368 bytes 2024-09-11T05:34:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1464750080 bytes 2024-09-11T05:34:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1482648576 bytes 2024-09-11T05:34:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1489399808 bytes 2024-09-11T05:34:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1545191424 bytes 2024-09-11T05:34:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1613577216 bytes 2024-09-11T05:34:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1624886272 bytes 2024-09-11T05:34:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1625532416 bytes 2024-09-11T05:34:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1722286080 bytes 2024-09-11T05:34:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1755085824 bytes 2024-09-11T05:34:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1774264320 bytes 2024-09-11T05:34:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1825560576 bytes 2024-09-11T05:34:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1838478336 bytes 2024-09-11T05:34:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1895216128 bytes 2024-09-11T05:34:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1963426816 bytes 2024-09-11T05:34:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1998539776 bytes 2024-09-11T05:34:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2074702848 bytes 2024-09-11T05:34:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2111038464 bytes 2024-09-11T05:34:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2179077120 bytes 2024-09-11T05:34:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2224314368 bytes 2024-09-11T05:34:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2250292224 bytes 2024-09-11T05:34:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2253233152 bytes 2024-09-11T05:34:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2288892928 bytes 2024-09-11T05:34:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2300997632 bytes 2024-09-11T05:34:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2385848320 bytes 2024-09-11T05:34:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2446781440 bytes 2024-09-11T05:34:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2461275136 bytes 2024-09-11T05:34:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2519952384 bytes 2024-09-11T05:34:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2545224704 bytes 2024-09-11T05:34:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2595520512 bytes 2024-09-11T05:34:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2629795840 bytes 2024-09-11T05:34:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2686236672 bytes 2024-09-11T05:34:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2686864384 bytes 2024-09-11T05:34:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2706513920 bytes 2024-09-11T05:34:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2818154496 bytes 2024-09-11T05:34:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2826881024 bytes 2024-09-11T05:34:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2931304448 bytes 2024-09-11T05:34:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 3026903040 bytes 2024-09-11T05:34:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3064788992 bytes 2024-09-11T05:34:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3132178432 bytes 2024-09-11T05:34:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3195080704 bytes 2024-09-11T05:34:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3232241664 bytes 2024-09-11T05:34:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3232851968 bytes 2024-09-11T05:34:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3282182144 bytes 2024-09-11T05:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3334988800 bytes 2024-09-11T05:34:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3455774720 bytes 2024-09-11T05:34:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3512946688 bytes 2024-09-11T05:34:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3546413056 bytes 2024-09-11T05:34:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3563426816 bytes 2024-09-11T05:34:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3611071488 bytes 2024-09-11T05:34:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3660107776 bytes 2024-09-11T05:34:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3710334976 bytes 2024-09-11T05:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3732968448 bytes 2024-09-11T05:34:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3789298688 bytes 2024-09-11T05:34:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3864897536 bytes 2024-09-11T05:34:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3936492544 bytes 2024-09-11T05:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:34:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:34:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:34:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:34:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:34:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17822993.07,"bytes_per_snapshot":533368832.0,"last_snapshot_bytes":534689792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032851,"remote_snapshot_timestamp":1726032851,"replay_state":"idle"} 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.1 on smithi164 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:34:22 2024-09-11T05:34:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:34:10' 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:34:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:34:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:34:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17822993.07,"bytes_per_snapshot":533368832.0,"last_snapshot_bytes":534689792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032851,"remote_snapshot_timestamp":1726032851,"replay_state":"idle"} 2024-09-11T05:34:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:34:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:34:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:34:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:34:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:34:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.1 on smithi164 2024-09-11T05:34:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:34:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:34:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:34:22' 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:34:22' 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:34:22' 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:34:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:34:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3etrueOK45484551mirror.2smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17822993.07,"bytes_per_snapshot":533368832.0,"last_snapshot_bytes":534689792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032851,"remote_snapshot_timestamp":1726032851,"replay_state":"idle"}44114414mirror.1smithi1642024-09-11 05:34:22cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:34:10' 2024-09-11T05:34:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:34:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:34:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:34:22' 2024-09-11T05:34:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:34:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:34:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:34:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap9 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2024-09-11T05:34:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:34:50.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9604691926253186) > weight (0.25) 2024-09-11T05:34:50.543 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.616668369719952) > weight (0.25) 2024-09-11T05:34:50.543 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-11T05:34:50.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-11T05:34:50.543 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:34:50.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T05:34:50.545+0000 7ff5a1e0d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 30271) UID: 1000 2024-09-11T05:34:56.544 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-11T05:34:56.544 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:34:56.545 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-11T05:34:56.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T05:34:56.545 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T05:34:56.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T05:34:56.548 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-11T05:35:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:35:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:35:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:35:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:35:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:35:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:35:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:35:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:35:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:35:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:35:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:35:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:35:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:35:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:35:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:35:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:35:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:35:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:35:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:35:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:35:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:35:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:35:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:35:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:35:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:35:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:35:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:35:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:35:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:35:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:35:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:35:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:35:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:35:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:35:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:35:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:35:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:35:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:35:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:35:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 29 2024-09-11T05:35:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:35:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=29 2024-09-11T05:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = '' 2024-09-11T05:35:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:35:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = '' 2024-09-11T05:35:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = '' 2024-09-11T05:35:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:35:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=29 2024-09-11T05:35:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = 29 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:35:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:35:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 18 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:35:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:34:18 2024 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:35:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:35:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap8-clean ']' 2024-09-11T05:35:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:35:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:35:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:35:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T05:35:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:35:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:35:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:35:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap9-clean 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2024-09-11T05:35:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:35:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:35:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T05:35:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:35:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:35:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:35:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T05:35:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 22343680 bytes 2024-09-11T05:35:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 45937664 bytes 2024-09-11T05:35:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 68895744 bytes 2024-09-11T05:35:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 119683072 bytes 2024-09-11T05:35:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 166482944 bytes 2024-09-11T05:35:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 262775808 bytes 2024-09-11T05:35:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 369890304 bytes 2024-09-11T05:35:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 431903744 bytes 2024-09-11T05:35:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 459568128 bytes 2024-09-11T05:35:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 509318144 bytes 2024-09-11T05:35:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 531751936 bytes 2024-09-11T05:35:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 532365312 bytes 2024-09-11T05:35:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 580800512 bytes 2024-09-11T05:35:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 630247424 bytes 2024-09-11T05:35:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 660504576 bytes 2024-09-11T05:35:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 697440256 bytes 2024-09-11T05:35:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 698058752 bytes 2024-09-11T05:35:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 703819776 bytes 2024-09-11T05:35:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 725082112 bytes 2024-09-11T05:35:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 756467712 bytes 2024-09-11T05:35:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 809702400 bytes 2024-09-11T05:35:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 866721792 bytes 2024-09-11T05:35:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 907551744 bytes 2024-09-11T05:35:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 982037504 bytes 2024-09-11T05:35:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1036382208 bytes 2024-09-11T05:35:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1071886336 bytes 2024-09-11T05:35:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1112071168 bytes 2024-09-11T05:35:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1156737024 bytes 2024-09-11T05:35:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1215850496 bytes 2024-09-11T05:35:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1289132032 bytes 2024-09-11T05:35:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1419380736 bytes 2024-09-11T05:35:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1501843456 bytes 2024-09-11T05:35:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1548851200 bytes 2024-09-11T05:35:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1576078336 bytes 2024-09-11T05:35:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1624893440 bytes 2024-09-11T05:35:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1726336000 bytes 2024-09-11T05:35:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1760801792 bytes 2024-09-11T05:35:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1795597312 bytes 2024-09-11T05:35:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1837824000 bytes 2024-09-11T05:35:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1872329728 bytes 2024-09-11T05:35:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1931454464 bytes 2024-09-11T05:35:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1994465280 bytes 2024-09-11T05:35:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2035588096 bytes 2024-09-11T05:35:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2081431552 bytes 2024-09-11T05:35:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2137996288 bytes 2024-09-11T05:35:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2220481536 bytes 2024-09-11T05:35:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2287391744 bytes 2024-09-11T05:35:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2306752512 bytes 2024-09-11T05:35:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2404146176 bytes 2024-09-11T05:35:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2443147264 bytes 2024-09-11T05:35:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2443760640 bytes 2024-09-11T05:35:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2512345088 bytes 2024-09-11T05:35:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2576125952 bytes 2024-09-11T05:35:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2586753024 bytes 2024-09-11T05:35:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2669179904 bytes 2024-09-11T05:35:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2742346752 bytes 2024-09-11T05:35:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2786262016 bytes 2024-09-11T05:35:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2830507008 bytes 2024-09-11T05:35:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:35:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:35:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:35:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:35:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2024-09-11T05:35:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:35:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:35:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='test: 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17874909.87,"bytes_per_snapshot":535468544.0,"last_snapshot_bytes":536247296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032906,"remote_snapshot_timestamp":1726032906,"replay_state":"idle"} 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.1 on smithi164 2024-09-11T05:35:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:35:22 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2024-09-11 05:35:05' 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T05:35:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2024-09-11T05:35:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^ description: .*' 2024-09-11T05:35:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17874909.87,"bytes_per_snapshot":535468544.0,"last_snapshot_bytes":536247296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032906,"remote_snapshot_timestamp":1726032906,"replay_state":"idle"} 2024-09-11T05:35:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:35:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:35:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep '^up+' 2024-09-11T05:35:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:35:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T05:35:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.1 on smithi164 2024-09-11T05:35:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update 2024-09-11T05:35:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:35:22' 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 05:35:22' 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2024-09-11T05:35:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local 'prev_last_update=2024-09-11 05:35:22' 2024-09-11T05:35:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status 2024-09-11T05:35:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ CEPH_ARGS= 2024-09-11T05:35:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T05:35:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status='OKOKOK142184266mirror.3smithi16419.1.1-298-gb53b6d3efalseOK43654368mirror.0smithi16419.1.1-298-gb53b6d3efalseOK44114414mirror.1smithi16419.1.1-298-gb53b6d3etrueOK45484551mirror.2smithi16419.1.1-298-gb53b6d3efalseOKtest38f5667d-f895-468e-984d-b354fbd0a1b4up+replayingreplaying, {"bytes_per_second":17874909.87,"bytes_per_snapshot":535468544.0,"last_snapshot_bytes":536247296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726032906,"remote_snapshot_timestamp":1726032906,"replay_state":"idle"}44114414mirror.1smithi1642024-09-11 05:35:22cluster275c5b592-5745-4d1a-984f-ff4a3feb0219up+stoppedlocal image is primary2024-09-11 05:35:05' 2024-09-11T05:35:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2024-09-11T05:35:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T05:35:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update='2024-09-11 05:35:22' 2024-09-11T05:35:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T05:35:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2024-09-11T05:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2024-09-11T05:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2024-09-11T05:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap10 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2024-09-11T05:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:35:41.534 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T05:35:41.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T05:35:41.535 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T05:35:41.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:35:41.536+0000 7ff35c190640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 31553) UID: 1000 2024-09-11T05:35:45.536 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T05:35:45.536 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:35:45.536 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T05:35:45.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T05:35:45.537 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T05:35:45.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T05:35:45.540 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:35:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:35:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:35:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:35:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:35:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:35:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:35:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:35:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:35:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:35:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:35:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:35:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:35:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:35:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:35:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:35:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:35:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:35:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:35:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:35:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:35:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:35:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:35:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:35:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:35:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:35:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:35:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:35:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:35:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:35:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:35:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:35:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:35:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:35:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:35:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:35:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:35:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:35:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:35:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:35:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:35:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:35:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:35:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:35:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:35:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:35:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:35:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:35:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:35:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:35:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:35:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:35:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:35:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:35:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:35:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:35:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:35:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:35:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 32 2024-09-11T05:35:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:35:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:35:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:35:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:35:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=32 2024-09-11T05:35:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:35:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 32 = '' 2024-09-11T05:35:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:35:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 32 = '' 2024-09-11T05:35:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:35:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:35:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:35:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:35:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:35:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:35:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:35:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:35:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:35:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:35:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:35:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:35:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:35:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 32 = '' 2024-09-11T05:35:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:35:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:36:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:36:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=32 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 32 = 32 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2024-09-11T05:36:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:36:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:36:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:36:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-11T05:36:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:36:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:36:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:36:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 20 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:34:18 2024 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:36:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:36:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snap9-clean ']' 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2024-09-11T05:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap10-clean 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2024-09-11T05:36:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T05:36:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:36:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:36:07.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3992767098115141) > weight (0.25) 2024-09-11T05:36:07.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5917531983806924) > weight (0.25) 2024-09-11T05:36:07.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7353001837555994) > weight (0.25) 2024-09-11T05:36:07.549 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T05:36:07.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T05:36:07.550 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:36:07.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:36:07.552+0000 7f502c8b9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 32207) UID: 1000 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:36:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:36:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T05:36:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:36:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:36:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:36:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:36:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:36:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:36:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T05:36:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:36:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:36:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:36:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:36:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:36:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:36:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:36:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:36:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:36:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:36:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T05:36:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T05:36:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T05:36:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2024-09-11T05:36:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:36:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:1 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:1 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=1 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:36:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:36:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=1 2024-09-11T05:36:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:36:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T05:36:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:1 2024-09-11T05:36:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:36:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=1 2024-09-11T05:36:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T05:36:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:36:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:36:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T05:36:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T05:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:36:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:36:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:36:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:36:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T05:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-test.status 2024-09-11T05:36:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:36:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:36:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T05:36:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-test.status 2024-09-11T05:36:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:36:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T05:36:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T05:36:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 34 2024-09-11T05:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.status 2024-09-11T05:36:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T05:36:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:36:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:36:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=34 2024-09-11T05:36:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:36:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:36:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:36:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:36:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:36:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:36:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:36:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:36:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:36:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:36:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:36:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 34 = '' 2024-09-11T05:36:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:36:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:36:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:36:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:36:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:36:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:36:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:36:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:36:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 34 = '' 2024-09-11T05:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:36:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:36:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.status 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T05:36:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:36:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=34 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 34 = 34 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:36:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:36:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-11T05:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2024-09-11T05:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 21 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a9c66db142 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a9c66db142 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:36:02 2024 2024-09-11T05:36:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:27:07 2024 2024-09-11T05:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 38f5667d-f895-468e-984d-b354fbd0a1b4 2024-09-11T05:36:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:36:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 10 2024-09-11T05:36:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2024-09-11T05:36:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T05:36:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:13.550 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T05:36:13.550 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:36:13.551 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T05:36:13.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:36:13.551 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:36:13.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:36:13.554 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-11T05:36:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1-clean 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2024-09-11T05:36:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2-clean 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3-clean 2024-09-11T05:36:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4-clean 2024-09-11T05:36:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T05:36:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2024-09-11T05:36:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5-clean 2024-09-11T05:36:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T05:36:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2024-09-11T05:36:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T05:36:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6-clean 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7-clean 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2024-09-11T05:36:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8-clean 2024-09-11T05:36:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2024-09-11T05:36:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9-clean 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2024-09-11T05:36:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2024-09-11T05:36:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10-clean 2024-09-11T05:36:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export 2024-09-11T05:36:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T05:36:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-test.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-test.export 2024-09-11T05:36:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:36:54.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 10 2024-09-11T05:36:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap1 2024-09-11T05:36:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-11T05:36:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:36:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1-clean 2024-09-11T05:36:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-11T05:36:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap1-clean 2024-09-11T05:36:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-11T05:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap2 2024-09-11T05:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-11T05:36:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2-clean 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap2-clean 2024-09-11T05:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-11T05:36:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap3 2024-09-11T05:36:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-11T05:36:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:36:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3-clean 2024-09-11T05:36:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-11T05:36:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:36:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:36:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:36:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap3-clean 2024-09-11T05:36:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-11T05:37:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap4 2024-09-11T05:37:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-11T05:37:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4-clean 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap4-clean 2024-09-11T05:37:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-11T05:37:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap5 2024-09-11T05:37:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-11T05:37:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5-clean 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap5-clean 2024-09-11T05:37:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-11T05:37:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap6 2024-09-11T05:37:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-11T05:37:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6-clean 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap6-clean 2024-09-11T05:37:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-11T05:37:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap7 2024-09-11T05:37:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-11T05:37:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7-clean 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap7-clean 2024-09-11T05:37:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-11T05:37:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap8 2024-09-11T05:37:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-11T05:37:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8-clean 2024-09-11T05:37:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-11T05:37:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap8-clean 2024-09-11T05:37:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-11T05:37:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap9 2024-09-11T05:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-11T05:37:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9-clean 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap9-clean 2024-09-11T05:37:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-11T05:37:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap10 2024-09-11T05:37:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-11T05:37:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10-clean 2024-09-11T05:37:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-11T05:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap10-clean 2024-09-11T05:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-11T05:37:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror test 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:37:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:37:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror test 2024-09-11T05:37:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-11T05:37:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-11T05:37:15.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:37:15.139+0000 7fde29ed5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556fe5765a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:37:15.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:37:15.139+0000 7fde306e2640 -1 rbd::mirror::ImageReplayer: 0x556fe5912280 [3/38f5667d-f895-468e-984d-b354fbd0a1b4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:37:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror test 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:37:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=test 2024-09-11T05:37:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T05:37:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:37:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T05:37:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:37:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:37:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:37:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-11T05:37:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:37:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror test 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=test 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T05:37:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:37:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T05:37:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: create many images' 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:37:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:37:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:37:17 TEST: create many images 2024-09-11T05:37:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:37:17 TEST: create many images 2024-09-11T05:37:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap 2024-09-11T05:37:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 50 2024-09-11T05:37:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:37:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2024-09-11T05:37:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:37:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-11T05:37:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:37:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-11T05:37:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 127837d7f38f 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.127837d7f38f 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:17 2024 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:17 2024 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:17 2024 2024-09-11T05:37:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:37:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:37:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0081d803-369d-4214-80b0-726959bc08b4 2024-09-11T05:37:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:37:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:37:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T05:37:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:37:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:37:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:37:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:37:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:37:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 58881024 bytes 2024-09-11T05:37:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 102090752 bytes 2024-09-11T05:37:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 142952448 bytes 2024-09-11T05:37:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 188137472 bytes 2024-09-11T05:37:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 239051776 bytes 2024-09-11T05:37:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 280876032 bytes 2024-09-11T05:37:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 321195008 bytes 2024-09-11T05:37:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 363832320 bytes 2024-09-11T05:37:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 442964992 bytes 2024-09-11T05:37:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 476984320 bytes 2024-09-11T05:37:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 522508288 bytes 2024-09-11T05:37:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 534367232 bytes 2024-09-11T05:37:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 590994432 bytes 2024-09-11T05:37:19.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9135090905349199) > weight (0.25) 2024-09-11T05:37:19.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.579157833065187) > weight (0.25) 2024-09-11T05:37:19.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.69715574083694) > weight (0.25) 2024-09-11T05:37:19.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5968383082438634) > weight (0.25) 2024-09-11T05:37:19.541 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-11T05:37:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 712593408 bytes 2024-09-11T05:37:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 734420992 bytes 2024-09-11T05:37:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 814527488 bytes 2024-09-11T05:37:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 848187392 bytes 2024-09-11T05:37:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 860116992 bytes 2024-09-11T05:37:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 978392064 bytes 2024-09-11T05:37:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1040495616 bytes 2024-09-11T05:37:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1096360960 bytes 2024-09-11T05:37:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1130173440 bytes 2024-09-11T05:37:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1169840128 bytes 2024-09-11T05:37:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1202328576 bytes 2024-09-11T05:37:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1202932736 bytes 2024-09-11T05:37:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1255768064 bytes 2024-09-11T05:37:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1295523840 bytes 2024-09-11T05:37:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1306293248 bytes 2024-09-11T05:37:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1386228736 bytes 2024-09-11T05:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1458527232 bytes 2024-09-11T05:37:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1470299136 bytes 2024-09-11T05:37:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1498902528 bytes 2024-09-11T05:37:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1531610112 bytes 2024-09-11T05:37:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1569888256 bytes 2024-09-11T05:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1615492096 bytes 2024-09-11T05:37:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1649024000 bytes 2024-09-11T05:37:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1712652288 bytes 2024-09-11T05:37:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1762702336 bytes 2024-09-11T05:37:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1779174400 bytes 2024-09-11T05:37:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1787688960 bytes 2024-09-11T05:37:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1837409280 bytes 2024-09-11T05:37:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1900590080 bytes 2024-09-11T05:37:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1944883200 bytes 2024-09-11T05:37:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2004232192 bytes 2024-09-11T05:37:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2080893952 bytes 2024-09-11T05:37:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2128442368 bytes 2024-09-11T05:37:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2168668160 bytes 2024-09-11T05:37:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2220413952 bytes 2024-09-11T05:37:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2256603136 bytes 2024-09-11T05:37:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2268819456 bytes 2024-09-11T05:37:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2317158400 bytes 2024-09-11T05:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2379331584 bytes 2024-09-11T05:37:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2396146688 bytes 2024-09-11T05:37:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2419378176 bytes 2024-09-11T05:37:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2433256448 bytes 2024-09-11T05:37:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2433876992 bytes 2024-09-11T05:37:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2484452352 bytes 2024-09-11T05:37:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:37:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:37:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:37:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:37:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:37:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-11T05:37:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:37:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:37:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-11T05:37:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:37:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-11T05:37:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:37:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-11T05:37:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2024-09-11T05:37:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:37:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1284f9154e48 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1284f9154e48 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:24 2024 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:24 2024 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:24 2024 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:37:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 8de57868-2d26-4418-9611-9fe7ca6aec5d 2024-09-11T05:37:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:37:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:37:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T05:37:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:37:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:37:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:37:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:37:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:37:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 23796736 bytes 2024-09-11T05:37:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 72678400 bytes 2024-09-11T05:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 158484480 bytes 2024-09-11T05:37:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 197762048 bytes 2024-09-11T05:37:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 206585856 bytes 2024-09-11T05:37:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 207208448 bytes 2024-09-11T05:37:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 219092992 bytes 2024-09-11T05:37:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 221504512 bytes 2024-09-11T05:37:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 281182208 bytes 2024-09-11T05:37:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 345172992 bytes 2024-09-11T05:37:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 374688768 bytes 2024-09-11T05:37:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 484429824 bytes 2024-09-11T05:37:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 491659264 bytes 2024-09-11T05:37:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 503405568 bytes 2024-09-11T05:37:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 586122240 bytes 2024-09-11T05:37:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 632485888 bytes 2024-09-11T05:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 676916224 bytes 2024-09-11T05:37:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 712409088 bytes 2024-09-11T05:37:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 748536832 bytes 2024-09-11T05:37:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 849979392 bytes 2024-09-11T05:37:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 853275648 bytes 2024-09-11T05:37:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 907699200 bytes 2024-09-11T05:37:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 946220032 bytes 2024-09-11T05:37:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 979297280 bytes 2024-09-11T05:37:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1008527360 bytes 2024-09-11T05:37:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1063580672 bytes 2024-09-11T05:37:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1124558848 bytes 2024-09-11T05:37:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1149173760 bytes 2024-09-11T05:37:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1168117760 bytes 2024-09-11T05:37:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1224536064 bytes 2024-09-11T05:37:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1269100544 bytes 2024-09-11T05:37:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1279569920 bytes 2024-09-11T05:37:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1292717056 bytes 2024-09-11T05:37:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1299963904 bytes 2024-09-11T05:37:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1363133440 bytes 2024-09-11T05:37:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1382754304 bytes 2024-09-11T05:37:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1473765376 bytes 2024-09-11T05:37:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1474374656 bytes 2024-09-11T05:37:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1506328576 bytes 2024-09-11T05:37:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1514473472 bytes 2024-09-11T05:37:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1540407296 bytes 2024-09-11T05:37:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1543049216 bytes 2024-09-11T05:37:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1587145728 bytes 2024-09-11T05:37:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1664698368 bytes 2024-09-11T05:37:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1772298240 bytes 2024-09-11T05:37:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1836915712 bytes 2024-09-11T05:37:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1955553280 bytes 2024-09-11T05:37:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2011447296 bytes 2024-09-11T05:37:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2055617536 bytes 2024-09-11T05:37:31.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2109827072 bytes 2024-09-11T05:37:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2184344576 bytes 2024-09-11T05:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2246942720 bytes 2024-09-11T05:37:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2291059712 bytes 2024-09-11T05:37:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2340489216 bytes 2024-09-11T05:37:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2391219200 bytes 2024-09-11T05:37:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2391824384 bytes 2024-09-11T05:37:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2407333888 bytes 2024-09-11T05:37:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2456801280 bytes 2024-09-11T05:37:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2469659648 bytes 2024-09-11T05:37:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2480789504 bytes 2024-09-11T05:37:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2539915264 bytes 2024-09-11T05:37:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2573025280 bytes 2024-09-11T05:37:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2609198080 bytes 2024-09-11T05:37:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2635001856 bytes 2024-09-11T05:37:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2644870144 bytes 2024-09-11T05:37:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2662177792 bytes 2024-09-11T05:37:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2673346560 bytes 2024-09-11T05:37:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2690954240 bytes 2024-09-11T05:37:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2736907264 bytes 2024-09-11T05:37:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2737519616 bytes 2024-09-11T05:37:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2777672704 bytes 2024-09-11T05:37:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2819733504 bytes 2024-09-11T05:37:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2922035200 bytes 2024-09-11T05:37:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2929760256 bytes 2024-09-11T05:37:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3008669696 bytes 2024-09-11T05:37:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3068969984 bytes 2024-09-11T05:37:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3092909056 bytes 2024-09-11T05:37:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3186681856 bytes 2024-09-11T05:37:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3262720000 bytes 2024-09-11T05:37:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3287686144 bytes 2024-09-11T05:37:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3334606848 bytes 2024-09-11T05:37:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3393057792 bytes 2024-09-11T05:37:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3438510080 bytes 2024-09-11T05:37:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3509788672 bytes 2024-09-11T05:37:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3528292352 bytes 2024-09-11T05:37:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3577791488 bytes 2024-09-11T05:37:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3578399744 bytes 2024-09-11T05:37:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3615112192 bytes 2024-09-11T05:37:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3664748544 bytes 2024-09-11T05:37:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3714947072 bytes 2024-09-11T05:37:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3762282496 bytes 2024-09-11T05:37:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3807881216 bytes 2024-09-11T05:37:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3855012864 bytes 2024-09-11T05:37:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3877736448 bytes 2024-09-11T05:37:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3972569088 bytes 2024-09-11T05:37:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4037968896 bytes 2024-09-11T05:37:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4070237184 bytes 2024-09-11T05:37:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4164617216 bytes 2024-09-11T05:37:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4200978432 bytes 2024-09-11T05:37:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4263799808 bytes 2024-09-11T05:37:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4379001856 bytes 2024-09-11T05:37:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4439622656 bytes 2024-09-11T05:37:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4478008320 bytes 2024-09-11T05:37:38.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4478653440 bytes 2024-09-11T05:37:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4566999040 bytes 2024-09-11T05:37:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4642904064 bytes 2024-09-11T05:37:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4705609728 bytes 2024-09-11T05:37:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10800 IOs, 4759133184 bytes 2024-09-11T05:37:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10900 IOs, 4810865664 bytes 2024-09-11T05:37:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11000 IOs, 4831427584 bytes 2024-09-11T05:37:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11100 IOs, 4852961280 bytes 2024-09-11T05:37:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11200 IOs, 4892697600 bytes 2024-09-11T05:37:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11300 IOs, 4933514240 bytes 2024-09-11T05:37:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11400 IOs, 4977656832 bytes 2024-09-11T05:37:39.555 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4879695311564247) > weight (0.25) 2024-09-11T05:37:39.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.32286736008296735) > weight (0.25) 2024-09-11T05:37:39.556 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2751215910269883) > weight (0.25) 2024-09-11T05:37:39.556 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T05:37:39.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T05:37:39.560 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T05:37:39.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:37:39.555+0000 7f3b62953640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 35766) UID: 1000 2024-09-11T05:37:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11500 IOs, 5053071360 bytes 2024-09-11T05:37:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:11600 IOs, 5123893248 bytes 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:37:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:37:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:37:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-11T05:37:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:37:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:37:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-11T05:37:40.560 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T05:37:40.561 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:37:40.561 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T05:37:40.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:37:40.561 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:37:40.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:37:40.565 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-11T05:37:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:37:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 128f129db1c0 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.128f129db1c0 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:39 2024 2024-09-11T05:37:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:39 2024 2024-09-11T05:37:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:39 2024 2024-09-11T05:37:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:37:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:37:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 938fc742-34c2-4279-9e72-cc118f790d8f 2024-09-11T05:37:40.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:37:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:37:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T05:37:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:37:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:37:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:37:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:37:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 34276352 bytes 2024-09-11T05:37:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 71670784 bytes 2024-09-11T05:37:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 97081344 bytes 2024-09-11T05:37:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 123576320 bytes 2024-09-11T05:37:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 124191744 bytes 2024-09-11T05:37:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 152719360 bytes 2024-09-11T05:37:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 216465408 bytes 2024-09-11T05:37:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 242440192 bytes 2024-09-11T05:37:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 298400768 bytes 2024-09-11T05:37:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 321183744 bytes 2024-09-11T05:37:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 360674304 bytes 2024-09-11T05:37:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 369568768 bytes 2024-09-11T05:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 445845504 bytes 2024-09-11T05:37:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 461097984 bytes 2024-09-11T05:37:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 490943488 bytes 2024-09-11T05:37:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 534371328 bytes 2024-09-11T05:37:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 558970880 bytes 2024-09-11T05:37:42.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 582897664 bytes 2024-09-11T05:37:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 647234560 bytes 2024-09-11T05:37:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 720947200 bytes 2024-09-11T05:37:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 767269888 bytes 2024-09-11T05:37:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 839971840 bytes 2024-09-11T05:37:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 871439360 bytes 2024-09-11T05:37:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 911996928 bytes 2024-09-11T05:37:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 960687104 bytes 2024-09-11T05:37:43.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1024644096 bytes 2024-09-11T05:37:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1086325760 bytes 2024-09-11T05:37:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1132319744 bytes 2024-09-11T05:37:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1168557056 bytes 2024-09-11T05:37:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1212920832 bytes 2024-09-11T05:37:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1275859968 bytes 2024-09-11T05:37:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1276662784 bytes 2024-09-11T05:37:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1305231360 bytes 2024-09-11T05:37:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1316134912 bytes 2024-09-11T05:37:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1386764288 bytes 2024-09-11T05:37:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1420111872 bytes 2024-09-11T05:37:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1528670208 bytes 2024-09-11T05:37:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1560773632 bytes 2024-09-11T05:37:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1596679168 bytes 2024-09-11T05:37:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1608609792 bytes 2024-09-11T05:37:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1613885440 bytes 2024-09-11T05:37:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1704701952 bytes 2024-09-11T05:37:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1798098944 bytes 2024-09-11T05:37:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1841708032 bytes 2024-09-11T05:37:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1863910400 bytes 2024-09-11T05:37:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1912550400 bytes 2024-09-11T05:37:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1929306112 bytes 2024-09-11T05:37:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1966067712 bytes 2024-09-11T05:37:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2030772224 bytes 2024-09-11T05:37:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2106427392 bytes 2024-09-11T05:37:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2107017216 bytes 2024-09-11T05:37:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2213039104 bytes 2024-09-11T05:37:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:37:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:37:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:37:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:37:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-11T05:37:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-11T05:37:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2024-09-11T05:37:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a3e6932c11 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a3e6932c11 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:46 2024 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:46 2024 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:46 2024 2024-09-11T05:37:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:37:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:37:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 222ccbb3-07f2-4daf-aca6-881dd616be7f 2024-09-11T05:37:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:37:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T05:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:37:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:37:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:37:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:37:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 34270208 bytes 2024-09-11T05:37:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 56392704 bytes 2024-09-11T05:37:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 110745600 bytes 2024-09-11T05:37:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 141755392 bytes 2024-09-11T05:37:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 167846912 bytes 2024-09-11T05:37:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 190566400 bytes 2024-09-11T05:37:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 198030336 bytes 2024-09-11T05:37:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 198654976 bytes 2024-09-11T05:37:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 231186432 bytes 2024-09-11T05:37:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 272273408 bytes 2024-09-11T05:37:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 312523776 bytes 2024-09-11T05:37:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 369837056 bytes 2024-09-11T05:37:49.253 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:37:49.250+0000 7f3afe201640 -1 reset not still connected to 0x5651e1f40c30 2024-09-11T05:37:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 393208832 bytes 2024-09-11T05:37:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 426801152 bytes 2024-09-11T05:37:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 428640256 bytes 2024-09-11T05:37:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 438975488 bytes 2024-09-11T05:37:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 462423040 bytes 2024-09-11T05:37:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 500334592 bytes 2024-09-11T05:37:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 518284288 bytes 2024-09-11T05:37:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 566115328 bytes 2024-09-11T05:37:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 593641472 bytes 2024-09-11T05:37:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 658708480 bytes 2024-09-11T05:37:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 694005760 bytes 2024-09-11T05:37:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 730034176 bytes 2024-09-11T05:37:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 768830464 bytes 2024-09-11T05:37:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 810371072 bytes 2024-09-11T05:37:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 874286080 bytes 2024-09-11T05:37:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 877490176 bytes 2024-09-11T05:37:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 883423232 bytes 2024-09-11T05:37:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 952597504 bytes 2024-09-11T05:37:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1014708224 bytes 2024-09-11T05:37:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1027745792 bytes 2024-09-11T05:37:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1067869184 bytes 2024-09-11T05:37:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1113003008 bytes 2024-09-11T05:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1163252736 bytes 2024-09-11T05:37:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1182344192 bytes 2024-09-11T05:37:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1228009472 bytes 2024-09-11T05:37:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1260234752 bytes 2024-09-11T05:37:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1295221760 bytes 2024-09-11T05:37:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1300744192 bytes 2024-09-11T05:37:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1352469504 bytes 2024-09-11T05:37:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1353099264 bytes 2024-09-11T05:37:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1366431744 bytes 2024-09-11T05:37:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1452298240 bytes 2024-09-11T05:37:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1479226368 bytes 2024-09-11T05:37:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1532946432 bytes 2024-09-11T05:37:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1564187648 bytes 2024-09-11T05:37:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1591847936 bytes 2024-09-11T05:37:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1623837696 bytes 2024-09-11T05:37:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1654237184 bytes 2024-09-11T05:37:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1671868416 bytes 2024-09-11T05:37:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1746381824 bytes 2024-09-11T05:37:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1774279680 bytes 2024-09-11T05:37:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1817119744 bytes 2024-09-11T05:37:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1869256704 bytes 2024-09-11T05:37:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 1873302528 bytes 2024-09-11T05:37:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 1918730240 bytes 2024-09-11T05:37:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 1992777728 bytes 2024-09-11T05:37:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2032057344 bytes 2024-09-11T05:37:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2060362752 bytes 2024-09-11T05:37:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2102724608 bytes 2024-09-11T05:37:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2169935872 bytes 2024-09-11T05:37:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2204879872 bytes 2024-09-11T05:37:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2220111872 bytes 2024-09-11T05:37:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2250486784 bytes 2024-09-11T05:37:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2267164672 bytes 2024-09-11T05:37:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2288376832 bytes 2024-09-11T05:37:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2340360192 bytes 2024-09-11T05:37:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2375534592 bytes 2024-09-11T05:37:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2402227200 bytes 2024-09-11T05:37:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2512012288 bytes 2024-09-11T05:37:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2576008192 bytes 2024-09-11T05:37:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2580117504 bytes 2024-09-11T05:37:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2615496704 bytes 2024-09-11T05:37:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2653067264 bytes 2024-09-11T05:37:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2663916544 bytes 2024-09-11T05:37:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 2721302528 bytes 2024-09-11T05:37:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 2721898496 bytes 2024-09-11T05:37:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 2741945344 bytes 2024-09-11T05:37:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 2775162880 bytes 2024-09-11T05:37:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 2843743232 bytes 2024-09-11T05:37:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 2918165504 bytes 2024-09-11T05:37:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 2951021568 bytes 2024-09-11T05:37:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3051436032 bytes 2024-09-11T05:37:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3053776896 bytes 2024-09-11T05:37:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3129993216 bytes 2024-09-11T05:37:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3214861312 bytes 2024-09-11T05:37:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3223502848 bytes 2024-09-11T05:37:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3308464128 bytes 2024-09-11T05:37:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3407821824 bytes 2024-09-11T05:37:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3480555520 bytes 2024-09-11T05:37:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3531801600 bytes 2024-09-11T05:37:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3546167296 bytes 2024-09-11T05:37:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3560638464 bytes 2024-09-11T05:37:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3634196480 bytes 2024-09-11T05:37:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3666114560 bytes 2024-09-11T05:37:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 3702758400 bytes 2024-09-11T05:37:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 3788810240 bytes 2024-09-11T05:37:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 3823923200 bytes 2024-09-11T05:37:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 3881969664 bytes 2024-09-11T05:37:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 3909204992 bytes 2024-09-11T05:37:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 3966041088 bytes 2024-09-11T05:37:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 3990039552 bytes 2024-09-11T05:38:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10400 IOs, 4028281856 bytes 2024-09-11T05:38:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10500 IOs, 4031380480 bytes 2024-09-11T05:38:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10600 IOs, 4090853376 bytes 2024-09-11T05:38:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10700 IOs, 4123919360 bytes 2024-09-11T05:38:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10800 IOs, 4169188352 bytes 2024-09-11T05:38:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10900 IOs, 4205696000 bytes 2024-09-11T05:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-11T05:38:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-11T05:38:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-11T05:38:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:38:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-11T05:38:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12af54196d8a 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12af54196d8a 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:00 2024 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:00 2024 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:00 2024 2024-09-11T05:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:38:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:38:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4109d16e-0d23-49cb-8378-af90cd53b465 2024-09-11T05:38:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:38:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:38:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T05:38:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:38:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:38:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T05:38:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:38:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:38:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 40099840 bytes 2024-09-11T05:38:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 48796672 bytes 2024-09-11T05:38:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 71239680 bytes 2024-09-11T05:38:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 96175104 bytes 2024-09-11T05:38:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 130972672 bytes 2024-09-11T05:38:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 162848768 bytes 2024-09-11T05:38:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 211960832 bytes 2024-09-11T05:38:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 218993664 bytes 2024-09-11T05:38:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 288942080 bytes 2024-09-11T05:38:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 299692032 bytes 2024-09-11T05:38:03.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 336655360 bytes 2024-09-11T05:38:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 366305280 bytes 2024-09-11T05:38:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 431646720 bytes 2024-09-11T05:38:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 485378048 bytes 2024-09-11T05:38:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 496377856 bytes 2024-09-11T05:38:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 538849280 bytes 2024-09-11T05:38:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 558308352 bytes 2024-09-11T05:38:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 624097280 bytes 2024-09-11T05:38:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 660423680 bytes 2024-09-11T05:38:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 695972864 bytes 2024-09-11T05:38:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 735794176 bytes 2024-09-11T05:38:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 772630528 bytes 2024-09-11T05:38:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 805319680 bytes 2024-09-11T05:38:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 824353792 bytes 2024-09-11T05:38:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 874661888 bytes 2024-09-11T05:38:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 952842240 bytes 2024-09-11T05:38:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 978231296 bytes 2024-09-11T05:38:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1002607616 bytes 2024-09-11T05:38:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1005019136 bytes 2024-09-11T05:38:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1026076672 bytes 2024-09-11T05:38:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1102999552 bytes 2024-09-11T05:38:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1139028992 bytes 2024-09-11T05:38:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1147460608 bytes 2024-09-11T05:38:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1214227456 bytes 2024-09-11T05:38:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1291756544 bytes 2024-09-11T05:38:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1324785664 bytes 2024-09-11T05:38:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1455629312 bytes 2024-09-11T05:38:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1469620224 bytes 2024-09-11T05:38:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1477528576 bytes 2024-09-11T05:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1506993152 bytes 2024-09-11T05:38:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1525685248 bytes 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2024-09-11T05:38:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-11T05:38:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-11T05:38:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:38:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12bb4cb85d43 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12bb4cb85d43 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:38:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:06 2024 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:06 2024 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:06 2024 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fafb8d11-da7d-4858-9912-7044d668ce0c 2024-09-11T05:38:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:38:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:38:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T05:38:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:38:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:38:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:38:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:38:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:38:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 10614784 bytes 2024-09-11T05:38:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 69047296 bytes 2024-09-11T05:38:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 89222144 bytes 2024-09-11T05:38:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 109554688 bytes 2024-09-11T05:38:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 175337472 bytes 2024-09-11T05:38:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 243438592 bytes 2024-09-11T05:38:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 317896704 bytes 2024-09-11T05:38:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 335106048 bytes 2024-09-11T05:38:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 358470656 bytes 2024-09-11T05:38:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 400268288 bytes 2024-09-11T05:38:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 429968384 bytes 2024-09-11T05:38:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 464023552 bytes 2024-09-11T05:38:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 485522432 bytes 2024-09-11T05:38:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 496133120 bytes 2024-09-11T05:38:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 504656896 bytes 2024-09-11T05:38:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 510095360 bytes 2024-09-11T05:38:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 586687488 bytes 2024-09-11T05:38:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 645486592 bytes 2024-09-11T05:38:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 691217408 bytes 2024-09-11T05:38:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 730770432 bytes 2024-09-11T05:38:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 746360832 bytes 2024-09-11T05:38:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 792265728 bytes 2024-09-11T05:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 834032640 bytes 2024-09-11T05:38:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 870158336 bytes 2024-09-11T05:38:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 911316992 bytes 2024-09-11T05:38:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 932766720 bytes 2024-09-11T05:38:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 978841600 bytes 2024-09-11T05:38:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1042147328 bytes 2024-09-11T05:38:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1091055616 bytes 2024-09-11T05:38:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1152427008 bytes 2024-09-11T05:38:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1153032192 bytes 2024-09-11T05:38:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1213287424 bytes 2024-09-11T05:38:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1222384640 bytes 2024-09-11T05:38:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1262576640 bytes 2024-09-11T05:38:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1290446848 bytes 2024-09-11T05:38:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1342401536 bytes 2024-09-11T05:38:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1446980608 bytes 2024-09-11T05:38:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1457169408 bytes 2024-09-11T05:38:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1488280576 bytes 2024-09-11T05:38:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1507503104 bytes 2024-09-11T05:38:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1544299520 bytes 2024-09-11T05:38:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1589620736 bytes 2024-09-11T05:38:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1674121216 bytes 2024-09-11T05:38:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1766405120 bytes 2024-09-11T05:38:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1813240832 bytes 2024-09-11T05:38:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1813853184 bytes 2024-09-11T05:38:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1854548992 bytes 2024-09-11T05:38:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1882564608 bytes 2024-09-11T05:38:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1926898688 bytes 2024-09-11T05:38:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1932689408 bytes 2024-09-11T05:38:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1937423360 bytes 2024-09-11T05:38:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1989605376 bytes 2024-09-11T05:38:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2039267328 bytes 2024-09-11T05:38:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2076696576 bytes 2024-09-11T05:38:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2096026624 bytes 2024-09-11T05:38:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2120192000 bytes 2024-09-11T05:38:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2149204992 bytes 2024-09-11T05:38:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2149851136 bytes 2024-09-11T05:38:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2212497408 bytes 2024-09-11T05:38:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2276584448 bytes 2024-09-11T05:38:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2279597056 bytes 2024-09-11T05:38:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2370530304 bytes 2024-09-11T05:38:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2373912576 bytes 2024-09-11T05:38:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2424855552 bytes 2024-09-11T05:38:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2475580416 bytes 2024-09-11T05:38:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2490493952 bytes 2024-09-11T05:38:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2557266944 bytes 2024-09-11T05:38:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2599696384 bytes 2024-09-11T05:38:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2603839488 bytes 2024-09-11T05:38:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2648400896 bytes 2024-09-11T05:38:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2695596032 bytes 2024-09-11T05:38:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2746168320 bytes 2024-09-11T05:38:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2779809792 bytes 2024-09-11T05:38:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2813307904 bytes 2024-09-11T05:38:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2861841408 bytes 2024-09-11T05:38:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2871606272 bytes 2024-09-11T05:38:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 2887389184 bytes 2024-09-11T05:38:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 2923066368 bytes 2024-09-11T05:38:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 2943706112 bytes 2024-09-11T05:38:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 2964731904 bytes 2024-09-11T05:38:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3050442752 bytes 2024-09-11T05:38:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3063151616 bytes 2024-09-11T05:38:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3096748032 bytes 2024-09-11T05:38:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3166784512 bytes 2024-09-11T05:38:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3217709056 bytes 2024-09-11T05:38:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3314365440 bytes 2024-09-11T05:38:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3328579584 bytes 2024-09-11T05:38:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3367292928 bytes 2024-09-11T05:38:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3480869888 bytes 2024-09-11T05:38:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3524185088 bytes 2024-09-11T05:38:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3545871360 bytes 2024-09-11T05:38:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3606083584 bytes 2024-09-11T05:38:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3635497984 bytes 2024-09-11T05:38:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3673665536 bytes 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-11T05:38:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-11T05:38:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-11T05:38:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:38:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-11T05:38:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2024-09-11T05:38:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:38:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:38:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12c71bf8af15 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12c71bf8af15 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:19 2024 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:19 2024 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:19 2024 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f6c4c95b-26d9-427d-9fd8-9b7fe2a1b185 2024-09-11T05:38:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:38:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:38:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T05:38:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:38:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:38:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:38:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:38:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:38:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 63783936 bytes 2024-09-11T05:38:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 133969920 bytes 2024-09-11T05:38:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 208918528 bytes 2024-09-11T05:38:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 216576000 bytes 2024-09-11T05:38:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 251354112 bytes 2024-09-11T05:38:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 276683776 bytes 2024-09-11T05:38:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 349905920 bytes 2024-09-11T05:38:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 390754304 bytes 2024-09-11T05:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 455385088 bytes 2024-09-11T05:38:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 478044160 bytes 2024-09-11T05:38:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 519747584 bytes 2024-09-11T05:38:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 541732864 bytes 2024-09-11T05:38:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 567905280 bytes 2024-09-11T05:38:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 650350592 bytes 2024-09-11T05:38:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 708391936 bytes 2024-09-11T05:38:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 747680768 bytes 2024-09-11T05:38:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 839164928 bytes 2024-09-11T05:38:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 933402624 bytes 2024-09-11T05:38:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 950463488 bytes 2024-09-11T05:38:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1018244096 bytes 2024-09-11T05:38:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1030105088 bytes 2024-09-11T05:38:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1081063424 bytes 2024-09-11T05:38:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1114021888 bytes 2024-09-11T05:38:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1152140288 bytes 2024-09-11T05:38:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1194521600 bytes 2024-09-11T05:38:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1202155520 bytes 2024-09-11T05:38:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1280926720 bytes 2024-09-11T05:38:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1341125632 bytes 2024-09-11T05:38:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1393810432 bytes 2024-09-11T05:38:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1472562176 bytes 2024-09-11T05:38:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1497851904 bytes 2024-09-11T05:38:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1502922752 bytes 2024-09-11T05:38:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1559335936 bytes 2024-09-11T05:38:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1582523392 bytes 2024-09-11T05:38:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1631798272 bytes 2024-09-11T05:38:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1675217920 bytes 2024-09-11T05:38:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1796114432 bytes 2024-09-11T05:38:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1864222720 bytes 2024-09-11T05:38:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1924329472 bytes 2024-09-11T05:38:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1956233216 bytes 2024-09-11T05:38:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1980989440 bytes 2024-09-11T05:38:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2011333632 bytes 2024-09-11T05:38:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2096020480 bytes 2024-09-11T05:38:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2117086208 bytes 2024-09-11T05:38:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2178629632 bytes 2024-09-11T05:38:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2210035712 bytes 2024-09-11T05:38:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2216056832 bytes 2024-09-11T05:38:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2264527872 bytes 2024-09-11T05:38:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2266234880 bytes 2024-09-11T05:38:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2306898944 bytes 2024-09-11T05:38:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2359174144 bytes 2024-09-11T05:38:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2381361152 bytes 2024-09-11T05:38:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2408200192 bytes 2024-09-11T05:38:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2412236800 bytes 2024-09-11T05:38:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2475340800 bytes 2024-09-11T05:38:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2542550016 bytes 2024-09-11T05:38:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2594247680 bytes 2024-09-11T05:38:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2632764416 bytes 2024-09-11T05:38:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2708623360 bytes 2024-09-11T05:38:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2733652992 bytes 2024-09-11T05:38:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2757766144 bytes 2024-09-11T05:38:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2802814976 bytes 2024-09-11T05:38:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2835690496 bytes 2024-09-11T05:38:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2903129088 bytes 2024-09-11T05:38:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2951361536 bytes 2024-09-11T05:38:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3026408448 bytes 2024-09-11T05:38:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3052266496 bytes 2024-09-11T05:38:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3109258240 bytes 2024-09-11T05:38:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3134593024 bytes 2024-09-11T05:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3166112768 bytes 2024-09-11T05:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3225071616 bytes 2024-09-11T05:38:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3251099648 bytes 2024-09-11T05:38:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3318585344 bytes 2024-09-11T05:38:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3371850752 bytes 2024-09-11T05:38:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3372639232 bytes 2024-09-11T05:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3411756032 bytes 2024-09-11T05:38:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3420524544 bytes 2024-09-11T05:38:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3445638144 bytes 2024-09-11T05:38:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3492059136 bytes 2024-09-11T05:38:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3516073984 bytes 2024-09-11T05:38:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3543518208 bytes 2024-09-11T05:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3610229760 bytes 2024-09-11T05:38:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3657487360 bytes 2024-09-11T05:38:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3692453888 bytes 2024-09-11T05:38:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3740867584 bytes 2024-09-11T05:38:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3777821696 bytes 2024-09-11T05:38:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3806355456 bytes 2024-09-11T05:38:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3868791808 bytes 2024-09-11T05:38:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3873263616 bytes 2024-09-11T05:38:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3916171264 bytes 2024-09-11T05:38:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3953210368 bytes 2024-09-11T05:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4036112384 bytes 2024-09-11T05:38:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-11T05:38:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-11T05:38:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:38:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-11T05:38:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12d3341c7e37 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12d3341c7e37 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:32 2024 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:32 2024 2024-09-11T05:38:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:32 2024 2024-09-11T05:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: ab674d81-64fd-4a72-a384-57c22ab78502 2024-09-11T05:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:38:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2024-09-11T05:38:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:38:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:38:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 40984576 bytes 2024-09-11T05:38:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 107871232 bytes 2024-09-11T05:38:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 164585472 bytes 2024-09-11T05:38:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 193611776 bytes 2024-09-11T05:38:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 231876608 bytes 2024-09-11T05:38:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 350515200 bytes 2024-09-11T05:38:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 387643392 bytes 2024-09-11T05:38:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 396863488 bytes 2024-09-11T05:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 456342528 bytes 2024-09-11T05:38:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 545334272 bytes 2024-09-11T05:38:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 613623808 bytes 2024-09-11T05:38:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 634840064 bytes 2024-09-11T05:38:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 642275328 bytes 2024-09-11T05:38:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 658568192 bytes 2024-09-11T05:38:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 754215936 bytes 2024-09-11T05:38:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 814366720 bytes 2024-09-11T05:38:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 864034816 bytes 2024-09-11T05:38:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 902820864 bytes 2024-09-11T05:38:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 947684352 bytes 2024-09-11T05:38:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 957810688 bytes 2024-09-11T05:38:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1018779648 bytes 2024-09-11T05:38:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1087315968 bytes 2024-09-11T05:38:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1100276736 bytes 2024-09-11T05:38:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1113182208 bytes 2024-09-11T05:38:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1147777024 bytes 2024-09-11T05:38:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1201848320 bytes 2024-09-11T05:38:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1207084032 bytes 2024-09-11T05:38:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1228430336 bytes 2024-09-11T05:38:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1261046784 bytes 2024-09-11T05:38:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1319428096 bytes 2024-09-11T05:38:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1347604480 bytes 2024-09-11T05:38:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1398864896 bytes 2024-09-11T05:38:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1474914304 bytes 2024-09-11T05:38:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1532363776 bytes 2024-09-11T05:38:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1618615296 bytes 2024-09-11T05:38:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1682864128 bytes 2024-09-11T05:38:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1686619136 bytes 2024-09-11T05:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1757452288 bytes 2024-09-11T05:38:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1770038272 bytes 2024-09-11T05:38:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1808372736 bytes 2024-09-11T05:38:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1869103104 bytes 2024-09-11T05:38:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1892025344 bytes 2024-09-11T05:38:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1987753984 bytes 2024-09-11T05:38:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2020228096 bytes 2024-09-11T05:38:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2074962944 bytes 2024-09-11T05:38:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2146704384 bytes 2024-09-11T05:38:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2176267264 bytes 2024-09-11T05:38:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2200830976 bytes 2024-09-11T05:38:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2248286208 bytes 2024-09-11T05:38:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2291154944 bytes 2024-09-11T05:38:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2344035328 bytes 2024-09-11T05:38:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2374659072 bytes 2024-09-11T05:38:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2448402432 bytes 2024-09-11T05:38:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2463785984 bytes 2024-09-11T05:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2548941824 bytes 2024-09-11T05:38:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2568085504 bytes 2024-09-11T05:38:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2607160320 bytes 2024-09-11T05:38:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2671469568 bytes 2024-09-11T05:38:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2672120832 bytes 2024-09-11T05:38:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2758676480 bytes 2024-09-11T05:38:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2820137984 bytes 2024-09-11T05:38:41.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5323411614302543) > weight (0.25) 2024-09-11T05:38:41.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7176575119025546) > weight (0.25) 2024-09-11T05:38:41.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.565425933725048) > weight (0.25) 2024-09-11T05:38:41.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7582400659663127) > weight (0.25) 2024-09-11T05:38:41.543 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-11T05:38:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2845402112 bytes 2024-09-11T05:38:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2887471104 bytes 2024-09-11T05:38:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2932978688 bytes 2024-09-11T05:38:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2953662464 bytes 2024-09-11T05:38:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3004668928 bytes 2024-09-11T05:38:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3090593792 bytes 2024-09-11T05:38:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3103582208 bytes 2024-09-11T05:38:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3105251328 bytes 2024-09-11T05:38:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3110593536 bytes 2024-09-11T05:38:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3185515520 bytes 2024-09-11T05:38:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3229432832 bytes 2024-09-11T05:38:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3354846208 bytes 2024-09-11T05:38:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3446117376 bytes 2024-09-11T05:38:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3457543168 bytes 2024-09-11T05:38:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3491379200 bytes 2024-09-11T05:38:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3569499136 bytes 2024-09-11T05:38:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3593458688 bytes 2024-09-11T05:38:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3658884096 bytes 2024-09-11T05:38:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3693022208 bytes 2024-09-11T05:38:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3709169664 bytes 2024-09-11T05:38:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3785504768 bytes 2024-09-11T05:38:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3808182272 bytes 2024-09-11T05:38:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3832604672 bytes 2024-09-11T05:38:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3862002688 bytes 2024-09-11T05:38:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:38:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-11T05:38:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-11T05:38:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:38:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12dfeaec249a 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12dfeaec249a 2024-09-11T05:38:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:44 2024 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:44 2024 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:44 2024 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 29554ac5-9f7f-47c5-8ec4-11f432ce8b61 2024-09-11T05:38:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:38:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:38:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:38:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:38:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:38:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 36509696 bytes 2024-09-11T05:38:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 80744448 bytes 2024-09-11T05:38:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 146034688 bytes 2024-09-11T05:38:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 223321088 bytes 2024-09-11T05:38:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 223974400 bytes 2024-09-11T05:38:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 274550784 bytes 2024-09-11T05:38:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 297012224 bytes 2024-09-11T05:38:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 339037184 bytes 2024-09-11T05:38:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 381391872 bytes 2024-09-11T05:38:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 403085312 bytes 2024-09-11T05:38:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 506943488 bytes 2024-09-11T05:38:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 558598144 bytes 2024-09-11T05:38:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 612996096 bytes 2024-09-11T05:38:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 635916288 bytes 2024-09-11T05:38:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 713996288 bytes 2024-09-11T05:38:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 745213952 bytes 2024-09-11T05:38:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 775666688 bytes 2024-09-11T05:38:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 815783936 bytes 2024-09-11T05:38:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 845273088 bytes 2024-09-11T05:38:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 872925184 bytes 2024-09-11T05:38:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 883863552 bytes 2024-09-11T05:38:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 980191232 bytes 2024-09-11T05:38:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1019211776 bytes 2024-09-11T05:38:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1021000704 bytes 2024-09-11T05:38:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1058155520 bytes 2024-09-11T05:38:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1108628480 bytes 2024-09-11T05:38:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1154823168 bytes 2024-09-11T05:38:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1206832128 bytes 2024-09-11T05:38:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1249506304 bytes 2024-09-11T05:38:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1250140160 bytes 2024-09-11T05:38:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1277799424 bytes 2024-09-11T05:38:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1317856256 bytes 2024-09-11T05:38:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1339526144 bytes 2024-09-11T05:38:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1403754496 bytes 2024-09-11T05:38:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1428258816 bytes 2024-09-11T05:38:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1447256064 bytes 2024-09-11T05:38:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1540715520 bytes 2024-09-11T05:38:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1602098176 bytes 2024-09-11T05:38:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1654333440 bytes 2024-09-11T05:38:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1702702080 bytes 2024-09-11T05:38:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1804125184 bytes 2024-09-11T05:38:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1848442880 bytes 2024-09-11T05:38:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1851486208 bytes 2024-09-11T05:38:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1873861632 bytes 2024-09-11T05:38:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1941026816 bytes 2024-09-11T05:38:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1986270208 bytes 2024-09-11T05:38:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2034253824 bytes 2024-09-11T05:38:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2122509312 bytes 2024-09-11T05:38:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2176990208 bytes 2024-09-11T05:38:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2219724800 bytes 2024-09-11T05:38:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2271468544 bytes 2024-09-11T05:38:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2347724800 bytes 2024-09-11T05:38:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2404721664 bytes 2024-09-11T05:38:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2449550336 bytes 2024-09-11T05:38:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2465522688 bytes 2024-09-11T05:38:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2506621952 bytes 2024-09-11T05:38:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2571830272 bytes 2024-09-11T05:38:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2665483264 bytes 2024-09-11T05:38:53.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2682096640 bytes 2024-09-11T05:38:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2691578880 bytes 2024-09-11T05:38:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2712452096 bytes 2024-09-11T05:38:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2774774784 bytes 2024-09-11T05:38:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2864676864 bytes 2024-09-11T05:38:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 3022285824 bytes 2024-09-11T05:38:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3070953472 bytes 2024-09-11T05:38:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3086531584 bytes 2024-09-11T05:38:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3115113472 bytes 2024-09-11T05:38:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3150466048 bytes 2024-09-11T05:38:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3166023680 bytes 2024-09-11T05:38:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3213683712 bytes 2024-09-11T05:38:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3229176832 bytes 2024-09-11T05:38:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3265492992 bytes 2024-09-11T05:38:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3266094080 bytes 2024-09-11T05:38:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3288111104 bytes 2024-09-11T05:38:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3313963008 bytes 2024-09-11T05:38:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3349655552 bytes 2024-09-11T05:38:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3405505536 bytes 2024-09-11T05:38:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3497630720 bytes 2024-09-11T05:38:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3543826432 bytes 2024-09-11T05:38:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3618477056 bytes 2024-09-11T05:38:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3656656896 bytes 2024-09-11T05:38:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3727878144 bytes 2024-09-11T05:38:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3741082624 bytes 2024-09-11T05:38:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3771241472 bytes 2024-09-11T05:38:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3851168768 bytes 2024-09-11T05:38:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3873347584 bytes 2024-09-11T05:38:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3890618368 bytes 2024-09-11T05:38:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3902408704 bytes 2024-09-11T05:38:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3921785856 bytes 2024-09-11T05:38:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3970695168 bytes 2024-09-11T05:38:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 4004000768 bytes 2024-09-11T05:38:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4035282944 bytes 2024-09-11T05:38:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4056837120 bytes 2024-09-11T05:38:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 4092368896 bytes 2024-09-11T05:38:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4152778752 bytes 2024-09-11T05:38:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4177580032 bytes 2024-09-11T05:38:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4253118464 bytes 2024-09-11T05:38:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4333366272 bytes 2024-09-11T05:38:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9900 IOs, 4375307264 bytes 2024-09-11T05:38:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10000 IOs, 4412025856 bytes 2024-09-11T05:38:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10100 IOs, 4440178688 bytes 2024-09-11T05:38:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10200 IOs, 4462208000 bytes 2024-09-11T05:38:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:10300 IOs, 4506921984 bytes 2024-09-11T05:38:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:38:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:38:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:38:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:38:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:38:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-11T05:39:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:00.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-11T05:39:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2024-09-11T05:39:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12ebddb30a32 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12ebddb30a32 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:59 2024 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:59 2024 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:59 2024 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 417b48dd-e251-40f1-872a-99ad8bf806db 2024-09-11T05:39:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T05:39:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:39:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:39:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 45571072 bytes 2024-09-11T05:39:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 149818368 bytes 2024-09-11T05:39:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 190818304 bytes 2024-09-11T05:39:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 219592704 bytes 2024-09-11T05:39:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 266798080 bytes 2024-09-11T05:39:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 335472640 bytes 2024-09-11T05:39:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 393112576 bytes 2024-09-11T05:39:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 393750528 bytes 2024-09-11T05:39:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 430896128 bytes 2024-09-11T05:39:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 493930496 bytes 2024-09-11T05:39:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 533590016 bytes 2024-09-11T05:39:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 648098816 bytes 2024-09-11T05:39:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 694647808 bytes 2024-09-11T05:39:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 749634560 bytes 2024-09-11T05:39:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 752781312 bytes 2024-09-11T05:39:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 781879296 bytes 2024-09-11T05:39:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 836362240 bytes 2024-09-11T05:39:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 965227520 bytes 2024-09-11T05:39:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1024060416 bytes 2024-09-11T05:39:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1089121280 bytes 2024-09-11T05:39:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1123944448 bytes 2024-09-11T05:39:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1135912960 bytes 2024-09-11T05:39:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1147960320 bytes 2024-09-11T05:39:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1180788736 bytes 2024-09-11T05:39:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1210813440 bytes 2024-09-11T05:39:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1256829952 bytes 2024-09-11T05:39:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1310255104 bytes 2024-09-11T05:39:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1416549376 bytes 2024-09-11T05:39:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1448025088 bytes 2024-09-11T05:39:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1494069248 bytes 2024-09-11T05:39:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1572810752 bytes 2024-09-11T05:39:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1599115264 bytes 2024-09-11T05:39:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1675968512 bytes 2024-09-11T05:39:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1716743168 bytes 2024-09-11T05:39:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1742889984 bytes 2024-09-11T05:39:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1761244160 bytes 2024-09-11T05:39:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1774297088 bytes 2024-09-11T05:39:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1793331200 bytes 2024-09-11T05:39:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1805251584 bytes 2024-09-11T05:39:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1834447872 bytes 2024-09-11T05:39:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1839047680 bytes 2024-09-11T05:39:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1907948544 bytes 2024-09-11T05:39:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1933508608 bytes 2024-09-11T05:39:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1971670016 bytes 2024-09-11T05:39:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2040743936 bytes 2024-09-11T05:39:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2078124032 bytes 2024-09-11T05:39:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2101431296 bytes 2024-09-11T05:39:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2158995456 bytes 2024-09-11T05:39:07.113 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:39:07.113+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b32f9450 2024-09-11T05:39:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2215641088 bytes 2024-09-11T05:39:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2235942912 bytes 2024-09-11T05:39:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2280210432 bytes 2024-09-11T05:39:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2331015168 bytes 2024-09-11T05:39:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2362484736 bytes 2024-09-11T05:39:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2390547456 bytes 2024-09-11T05:39:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2433857536 bytes 2024-09-11T05:39:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2462004224 bytes 2024-09-11T05:39:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2468494336 bytes 2024-09-11T05:39:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2517265408 bytes 2024-09-11T05:39:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2584223744 bytes 2024-09-11T05:39:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2670725120 bytes 2024-09-11T05:39:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2765669376 bytes 2024-09-11T05:39:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2835166208 bytes 2024-09-11T05:39:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2879302656 bytes 2024-09-11T05:39:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2917691392 bytes 2024-09-11T05:39:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2943239168 bytes 2024-09-11T05:39:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2952925184 bytes 2024-09-11T05:39:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2953574400 bytes 2024-09-11T05:39:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2989033472 bytes 2024-09-11T05:39:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3004670976 bytes 2024-09-11T05:39:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3044034560 bytes 2024-09-11T05:39:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3071323136 bytes 2024-09-11T05:39:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3098233856 bytes 2024-09-11T05:39:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3135463424 bytes 2024-09-11T05:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3145220096 bytes 2024-09-11T05:39:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3182797824 bytes 2024-09-11T05:39:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3205937152 bytes 2024-09-11T05:39:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3247186944 bytes 2024-09-11T05:39:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3306427392 bytes 2024-09-11T05:39:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3345624064 bytes 2024-09-11T05:39:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3384796160 bytes 2024-09-11T05:39:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3396731904 bytes 2024-09-11T05:39:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3492953088 bytes 2024-09-11T05:39:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3558390784 bytes 2024-09-11T05:39:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3642243072 bytes 2024-09-11T05:39:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3695514624 bytes 2024-09-11T05:39:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3740187648 bytes 2024-09-11T05:39:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3776650240 bytes 2024-09-11T05:39:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3796362240 bytes 2024-09-11T05:39:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3796975616 bytes 2024-09-11T05:39:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3866089472 bytes 2024-09-11T05:39:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3919604736 bytes 2024-09-11T05:39:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3998683136 bytes 2024-09-11T05:39:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4112351232 bytes 2024-09-11T05:39:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 4129037312 bytes 2024-09-11T05:39:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4162089984 bytes 2024-09-11T05:39:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4181013504 bytes 2024-09-11T05:39:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4224063488 bytes 2024-09-11T05:39:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:39:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-11T05:39:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:39:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:39:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-11T05:39:14.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6374051871302489) > weight (0.25) 2024-09-11T05:39:14.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7195451578267085) > weight (0.25) 2024-09-11T05:39:14.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9847596887891402) > weight (0.25) 2024-09-11T05:39:14.567 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9288573000627716) > weight (0.25) 2024-09-11T05:39:14.567 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-11T05:39:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_11': 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12f7316ebf76 2024-09-11T05:39:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12f7316ebf76 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:13 2024 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:13 2024 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:13 2024 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f5febb24-4281-4e75-9b77-08bba442bf19 2024-09-11T05:39:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-11T05:39:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:39:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:39:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 54188032 bytes 2024-09-11T05:39:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 58611712 bytes 2024-09-11T05:39:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 89815040 bytes 2024-09-11T05:39:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 140073984 bytes 2024-09-11T05:39:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 174900224 bytes 2024-09-11T05:39:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 260290560 bytes 2024-09-11T05:39:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 289471488 bytes 2024-09-11T05:39:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 322544640 bytes 2024-09-11T05:39:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 327479296 bytes 2024-09-11T05:39:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 345186304 bytes 2024-09-11T05:39:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 375802880 bytes 2024-09-11T05:39:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 454665216 bytes 2024-09-11T05:39:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 509175808 bytes 2024-09-11T05:39:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 555262976 bytes 2024-09-11T05:39:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 645182464 bytes 2024-09-11T05:39:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 700097536 bytes 2024-09-11T05:39:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 772742144 bytes 2024-09-11T05:39:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 807856128 bytes 2024-09-11T05:39:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 856337408 bytes 2024-09-11T05:39:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 902709248 bytes 2024-09-11T05:39:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 932373504 bytes 2024-09-11T05:39:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 941588480 bytes 2024-09-11T05:39:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 977860608 bytes 2024-09-11T05:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1005360128 bytes 2024-09-11T05:39:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1047462912 bytes 2024-09-11T05:39:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1074653184 bytes 2024-09-11T05:39:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1088164864 bytes 2024-09-11T05:39:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1138025472 bytes 2024-09-11T05:39:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1200416768 bytes 2024-09-11T05:39:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1280436224 bytes 2024-09-11T05:39:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1306793984 bytes 2024-09-11T05:39:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1343728640 bytes 2024-09-11T05:39:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1425790976 bytes 2024-09-11T05:39:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1503800320 bytes 2024-09-11T05:39:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1577086976 bytes 2024-09-11T05:39:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1612817408 bytes 2024-09-11T05:39:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1659652096 bytes 2024-09-11T05:39:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1702775808 bytes 2024-09-11T05:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1773341696 bytes 2024-09-11T05:39:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1829579776 bytes 2024-09-11T05:39:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1858264064 bytes 2024-09-11T05:39:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1858890752 bytes 2024-09-11T05:39:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1913733120 bytes 2024-09-11T05:39:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1984905216 bytes 2024-09-11T05:39:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2033539072 bytes 2024-09-11T05:39:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2145709056 bytes 2024-09-11T05:39:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2176606208 bytes 2024-09-11T05:39:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2177222656 bytes 2024-09-11T05:39:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2234415104 bytes 2024-09-11T05:39:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2235018240 bytes 2024-09-11T05:39:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2294971392 bytes 2024-09-11T05:39:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2347741184 bytes 2024-09-11T05:39:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2367416320 bytes 2024-09-11T05:39:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2476244992 bytes 2024-09-11T05:39:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2512801792 bytes 2024-09-11T05:39:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2523147264 bytes 2024-09-11T05:39:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2572115968 bytes 2024-09-11T05:39:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2592598016 bytes 2024-09-11T05:39:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2681246720 bytes 2024-09-11T05:39:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2720760832 bytes 2024-09-11T05:39:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2744313856 bytes 2024-09-11T05:39:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2770002944 bytes 2024-09-11T05:39:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2881696768 bytes 2024-09-11T05:39:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2944591872 bytes 2024-09-11T05:39:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3013334016 bytes 2024-09-11T05:39:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3096418304 bytes 2024-09-11T05:39:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3136107520 bytes 2024-09-11T05:39:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3169739776 bytes 2024-09-11T05:39:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3220521984 bytes 2024-09-11T05:39:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3245070336 bytes 2024-09-11T05:39:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3291086848 bytes 2024-09-11T05:39:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3319512064 bytes 2024-09-11T05:39:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3352245248 bytes 2024-09-11T05:39:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3386738688 bytes 2024-09-11T05:39:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3416467456 bytes 2024-09-11T05:39:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3449787392 bytes 2024-09-11T05:39:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3507213312 bytes 2024-09-11T05:39:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3585631232 bytes 2024-09-11T05:39:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3639971840 bytes 2024-09-11T05:39:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3706371072 bytes 2024-09-11T05:39:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3739463680 bytes 2024-09-11T05:39:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3740051456 bytes 2024-09-11T05:39:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3753701376 bytes 2024-09-11T05:39:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3826178048 bytes 2024-09-11T05:39:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3838425088 bytes 2024-09-11T05:39:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3897990144 bytes 2024-09-11T05:39:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3978172416 bytes 2024-09-11T05:39:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 4046487552 bytes 2024-09-11T05:39:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 4055163904 bytes 2024-09-11T05:39:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 4117111808 bytes 2024-09-11T05:39:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 4149652480 bytes 2024-09-11T05:39:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4198448128 bytes 2024-09-11T05:39:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4239781888 bytes 2024-09-11T05:39:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 4240401408 bytes 2024-09-11T05:39:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4261305344 bytes 2024-09-11T05:39:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4306822144 bytes 2024-09-11T05:39:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 4343154688 bytes 2024-09-11T05:39:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 4370414592 bytes 2024-09-11T05:39:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:39:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:39:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:39:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-11T05:39:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_12': 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 130330ca30de 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.130330ca30de 2024-09-11T05:39:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:27 2024 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:27 2024 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:27 2024 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b16094f1-b9fc-4dcb-bceb-0331cdd63e45 2024-09-11T05:39:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-11T05:39:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:39:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:39:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 35579904 bytes 2024-09-11T05:39:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 52648960 bytes 2024-09-11T05:39:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 110528512 bytes 2024-09-11T05:39:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 174995456 bytes 2024-09-11T05:39:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 254046208 bytes 2024-09-11T05:39:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 278384640 bytes 2024-09-11T05:39:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 334979072 bytes 2024-09-11T05:39:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 351142912 bytes 2024-09-11T05:39:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 357321728 bytes 2024-09-11T05:39:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 434131968 bytes 2024-09-11T05:39:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 496422912 bytes 2024-09-11T05:39:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 542968832 bytes 2024-09-11T05:39:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 568452096 bytes 2024-09-11T05:39:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 643099648 bytes 2024-09-11T05:39:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 662001664 bytes 2024-09-11T05:39:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 737990656 bytes 2024-09-11T05:39:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 776281088 bytes 2024-09-11T05:39:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 779310080 bytes 2024-09-11T05:39:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 809823232 bytes 2024-09-11T05:39:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 837882880 bytes 2024-09-11T05:39:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 885608448 bytes 2024-09-11T05:39:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 917650432 bytes 2024-09-11T05:39:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 957972480 bytes 2024-09-11T05:39:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1046273024 bytes 2024-09-11T05:39:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1063928832 bytes 2024-09-11T05:39:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1107874816 bytes 2024-09-11T05:39:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1120927744 bytes 2024-09-11T05:39:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1160670208 bytes 2024-09-11T05:39:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1203665920 bytes 2024-09-11T05:39:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1214993408 bytes 2024-09-11T05:39:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1238844416 bytes 2024-09-11T05:39:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1255441408 bytes 2024-09-11T05:39:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1274858496 bytes 2024-09-11T05:39:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1294057472 bytes 2024-09-11T05:39:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1318632448 bytes 2024-09-11T05:39:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1399330816 bytes 2024-09-11T05:39:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1424606208 bytes 2024-09-11T05:39:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1454676992 bytes 2024-09-11T05:39:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1520559104 bytes 2024-09-11T05:39:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1579458560 bytes 2024-09-11T05:39:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1631619072 bytes 2024-09-11T05:39:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1672131584 bytes 2024-09-11T05:39:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1709523968 bytes 2024-09-11T05:39:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1731394560 bytes 2024-09-11T05:39:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1748744192 bytes 2024-09-11T05:39:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1789210624 bytes 2024-09-11T05:39:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1824581632 bytes 2024-09-11T05:39:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1845530624 bytes 2024-09-11T05:39:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1870433280 bytes 2024-09-11T05:39:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1888109568 bytes 2024-09-11T05:39:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1913296896 bytes 2024-09-11T05:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1935910912 bytes 2024-09-11T05:39:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2049884160 bytes 2024-09-11T05:39:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2123298816 bytes 2024-09-11T05:39:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2184515584 bytes 2024-09-11T05:39:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2198901760 bytes 2024-09-11T05:39:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2206885888 bytes 2024-09-11T05:39:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2275599360 bytes 2024-09-11T05:39:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2340137984 bytes 2024-09-11T05:39:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2418672640 bytes 2024-09-11T05:39:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2477137920 bytes 2024-09-11T05:39:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2506378240 bytes 2024-09-11T05:39:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2571019264 bytes 2024-09-11T05:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2571651072 bytes 2024-09-11T05:39:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2603248640 bytes 2024-09-11T05:39:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2654767104 bytes 2024-09-11T05:39:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2742367232 bytes 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:39:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:39:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:39:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-11T05:39:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-11T05:39:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-11T05:39:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_13': 2024-09-11T05:39:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 130fb084920 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.130fb084920 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:37 2024 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:37 2024 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:37 2024 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a98d5b12-6ee4-49e3-b583-ddcaf0392d73 2024-09-11T05:39:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-11T05:39:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:39:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:39:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 33718272 bytes 2024-09-11T05:39:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 58975232 bytes 2024-09-11T05:39:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 97982464 bytes 2024-09-11T05:39:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 170478592 bytes 2024-09-11T05:39:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 194488320 bytes 2024-09-11T05:39:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 319451136 bytes 2024-09-11T05:39:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 363141120 bytes 2024-09-11T05:39:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 421117952 bytes 2024-09-11T05:39:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 468869120 bytes 2024-09-11T05:39:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 502340608 bytes 2024-09-11T05:39:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 556549120 bytes 2024-09-11T05:39:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 577142784 bytes 2024-09-11T05:39:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 577775616 bytes 2024-09-11T05:39:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 617256960 bytes 2024-09-11T05:39:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 663835648 bytes 2024-09-11T05:39:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 742152192 bytes 2024-09-11T05:39:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 790035456 bytes 2024-09-11T05:39:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 846797824 bytes 2024-09-11T05:39:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 877761536 bytes 2024-09-11T05:39:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 960764928 bytes 2024-09-11T05:39:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 996344832 bytes 2024-09-11T05:39:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1022107648 bytes 2024-09-11T05:39:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1068667904 bytes 2024-09-11T05:39:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1132444672 bytes 2024-09-11T05:39:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1222081536 bytes 2024-09-11T05:39:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1259101184 bytes 2024-09-11T05:39:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1273669632 bytes 2024-09-11T05:39:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1344494592 bytes 2024-09-11T05:39:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1394545664 bytes 2024-09-11T05:39:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1476601856 bytes 2024-09-11T05:39:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1484855296 bytes 2024-09-11T05:39:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1558110208 bytes 2024-09-11T05:39:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1620892672 bytes 2024-09-11T05:39:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1640064000 bytes 2024-09-11T05:39:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1677887488 bytes 2024-09-11T05:39:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1780455424 bytes 2024-09-11T05:39:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1789611008 bytes 2024-09-11T05:39:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1826065408 bytes 2024-09-11T05:39:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1863788544 bytes 2024-09-11T05:39:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1914076160 bytes 2024-09-11T05:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1951341568 bytes 2024-09-11T05:39:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2023620608 bytes 2024-09-11T05:39:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2046415872 bytes 2024-09-11T05:39:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2083418112 bytes 2024-09-11T05:39:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2147675136 bytes 2024-09-11T05:39:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2206489600 bytes 2024-09-11T05:39:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2245920768 bytes 2024-09-11T05:39:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2283376640 bytes 2024-09-11T05:39:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2322186240 bytes 2024-09-11T05:39:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2333535232 bytes 2024-09-11T05:39:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2401822720 bytes 2024-09-11T05:39:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2445197312 bytes 2024-09-11T05:39:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2472228864 bytes 2024-09-11T05:39:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2495893504 bytes 2024-09-11T05:39:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2519105536 bytes 2024-09-11T05:39:46.543 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6278708023647404) > weight (0.25) 2024-09-11T05:39:46.543 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8504079491914187) > weight (0.25) 2024-09-11T05:39:46.544 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5032622565929724) > weight (0.25) 2024-09-11T05:39:46.544 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.27198185562037236) > weight (0.25) 2024-09-11T05:39:46.544 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-11T05:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2588288000 bytes 2024-09-11T05:39:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2622398464 bytes 2024-09-11T05:39:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2657796096 bytes 2024-09-11T05:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2707762176 bytes 2024-09-11T05:39:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2788115456 bytes 2024-09-11T05:39:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2830283776 bytes 2024-09-11T05:39:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2846819328 bytes 2024-09-11T05:39:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2861570048 bytes 2024-09-11T05:39:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2910259200 bytes 2024-09-11T05:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-11T05:39:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:39:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-11T05:39:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_14': 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 131b32e6eb33 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.131b32e6eb33 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:47 2024 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:47 2024 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:47 2024 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5e3847d7-8152-4a70-94e2-5fa8963bddcf 2024-09-11T05:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-11T05:39:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 27387904 bytes 2024-09-11T05:39:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 65874944 bytes 2024-09-11T05:39:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 141436928 bytes 2024-09-11T05:39:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 198623232 bytes 2024-09-11T05:39:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 268493824 bytes 2024-09-11T05:39:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 343475200 bytes 2024-09-11T05:39:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 372443136 bytes 2024-09-11T05:39:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 426629120 bytes 2024-09-11T05:39:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 473468928 bytes 2024-09-11T05:39:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 519649280 bytes 2024-09-11T05:39:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 551654400 bytes 2024-09-11T05:39:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 637467648 bytes 2024-09-11T05:39:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 710430720 bytes 2024-09-11T05:39:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 797562880 bytes 2024-09-11T05:39:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 885083136 bytes 2024-09-11T05:39:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 922587136 bytes 2024-09-11T05:39:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 1019674624 bytes 2024-09-11T05:39:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 1116783616 bytes 2024-09-11T05:39:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1158002688 bytes 2024-09-11T05:39:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1237308416 bytes 2024-09-11T05:39:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1293106176 bytes 2024-09-11T05:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1355809792 bytes 2024-09-11T05:39:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1382979584 bytes 2024-09-11T05:39:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1447570432 bytes 2024-09-11T05:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1506502656 bytes 2024-09-11T05:39:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1553998848 bytes 2024-09-11T05:39:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1578229760 bytes 2024-09-11T05:39:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1599292416 bytes 2024-09-11T05:39:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1661043712 bytes 2024-09-11T05:39:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1713439744 bytes 2024-09-11T05:39:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1798236160 bytes 2024-09-11T05:39:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1838483456 bytes 2024-09-11T05:39:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1900519424 bytes 2024-09-11T05:39:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1921432576 bytes 2024-09-11T05:39:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 2021407744 bytes 2024-09-11T05:39:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 2054230016 bytes 2024-09-11T05:39:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 2089591808 bytes 2024-09-11T05:39:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 2138170368 bytes 2024-09-11T05:39:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 2200349696 bytes 2024-09-11T05:39:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 2242606080 bytes 2024-09-11T05:39:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:39:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:39:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:39:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:39:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:39:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-11T05:39:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:39:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:39:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-11T05:39:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:39:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_15': 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1327fa795cba 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1327fa795cba 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:54 2024 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:54 2024 2024-09-11T05:39:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:54 2024 2024-09-11T05:39:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:39:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:39:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 61ff442c-b39c-42e8-bcaf-1b8c56417011 2024-09-11T05:39:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:39:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:39:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-11T05:39:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:39:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:39:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:39:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:39:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:39:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:39:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:39:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 53602304 bytes 2024-09-11T05:39:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 92933120 bytes 2024-09-11T05:39:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 109508608 bytes 2024-09-11T05:39:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 234226688 bytes 2024-09-11T05:39:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 312733696 bytes 2024-09-11T05:39:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 347148288 bytes 2024-09-11T05:39:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 423771136 bytes 2024-09-11T05:39:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 442835968 bytes 2024-09-11T05:39:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 521050112 bytes 2024-09-11T05:39:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 521655296 bytes 2024-09-11T05:39:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 557535232 bytes 2024-09-11T05:39:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 566410240 bytes 2024-09-11T05:39:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 588720128 bytes 2024-09-11T05:39:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 640166912 bytes 2024-09-11T05:39:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 728077312 bytes 2024-09-11T05:39:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 796437504 bytes 2024-09-11T05:39:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 880572416 bytes 2024-09-11T05:39:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 918566912 bytes 2024-09-11T05:39:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 934164480 bytes 2024-09-11T05:39:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1003387904 bytes 2024-09-11T05:39:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1019264000 bytes 2024-09-11T05:39:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1019914240 bytes 2024-09-11T05:39:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1090145280 bytes 2024-09-11T05:39:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1130908672 bytes 2024-09-11T05:39:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1170487296 bytes 2024-09-11T05:39:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1218700288 bytes 2024-09-11T05:39:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1263971328 bytes 2024-09-11T05:39:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1310752768 bytes 2024-09-11T05:40:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1390212096 bytes 2024-09-11T05:40:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1418917888 bytes 2024-09-11T05:40:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1474598912 bytes 2024-09-11T05:40:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1503230976 bytes 2024-09-11T05:40:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1541438464 bytes 2024-09-11T05:40:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1601742848 bytes 2024-09-11T05:40:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1640925184 bytes 2024-09-11T05:40:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1694517248 bytes 2024-09-11T05:40:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1757829120 bytes 2024-09-11T05:40:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1801185280 bytes 2024-09-11T05:40:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1842872320 bytes 2024-09-11T05:40:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1872974848 bytes 2024-09-11T05:40:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1897141248 bytes 2024-09-11T05:40:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1951630336 bytes 2024-09-11T05:40:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1991239680 bytes 2024-09-11T05:40:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2010011648 bytes 2024-09-11T05:40:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2061774848 bytes 2024-09-11T05:40:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2087494656 bytes 2024-09-11T05:40:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2146105344 bytes 2024-09-11T05:40:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2186193920 bytes 2024-09-11T05:40:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2215102464 bytes 2024-09-11T05:40:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2245781504 bytes 2024-09-11T05:40:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2246391808 bytes 2024-09-11T05:40:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2290494464 bytes 2024-09-11T05:40:03.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2344714240 bytes 2024-09-11T05:40:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2355415040 bytes 2024-09-11T05:40:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2389293056 bytes 2024-09-11T05:40:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2478856192 bytes 2024-09-11T05:40:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2528132096 bytes 2024-09-11T05:40:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2554434560 bytes 2024-09-11T05:40:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2630813696 bytes 2024-09-11T05:40:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2682961920 bytes 2024-09-11T05:40:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2720065536 bytes 2024-09-11T05:40:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2761120768 bytes 2024-09-11T05:40:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2779440128 bytes 2024-09-11T05:40:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2811856896 bytes 2024-09-11T05:40:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2838216704 bytes 2024-09-11T05:40:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2843240448 bytes 2024-09-11T05:40:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2888279040 bytes 2024-09-11T05:40:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2898213888 bytes 2024-09-11T05:40:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2928353280 bytes 2024-09-11T05:40:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2961561600 bytes 2024-09-11T05:40:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3014942720 bytes 2024-09-11T05:40:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3088739328 bytes 2024-09-11T05:40:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3109198848 bytes 2024-09-11T05:40:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3201786880 bytes 2024-09-11T05:40:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3213932544 bytes 2024-09-11T05:40:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3272304640 bytes 2024-09-11T05:40:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3318322176 bytes 2024-09-11T05:40:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3343917056 bytes 2024-09-11T05:40:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3377804288 bytes 2024-09-11T05:40:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3415768064 bytes 2024-09-11T05:40:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3444430848 bytes 2024-09-11T05:40:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3502502912 bytes 2024-09-11T05:40:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3528517632 bytes 2024-09-11T05:40:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3570198528 bytes 2024-09-11T05:40:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3596620800 bytes 2024-09-11T05:40:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3636493312 bytes 2024-09-11T05:40:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3712960512 bytes 2024-09-11T05:40:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3756465152 bytes 2024-09-11T05:40:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3833263104 bytes 2024-09-11T05:40:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3876523008 bytes 2024-09-11T05:40:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3956222976 bytes 2024-09-11T05:40:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4029698048 bytes 2024-09-11T05:40:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4039913472 bytes 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2024-09-11T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-11T05:40:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:40:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-11T05:40:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:40:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-11T05:40:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_16': 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1333c9d573fe 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1333c9d573fe 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:40:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:09 2024 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:09 2024 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:09 2024 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e19e65a3-67e6-4b67-bbe6-97b140e081a0 2024-09-11T05:40:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:40:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:40:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-11T05:40:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:40:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:40:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:40:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:40:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:40:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 54914048 bytes 2024-09-11T05:40:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 94401536 bytes 2024-09-11T05:40:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 160064512 bytes 2024-09-11T05:40:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 211704832 bytes 2024-09-11T05:40:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 235295744 bytes 2024-09-11T05:40:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 244956160 bytes 2024-09-11T05:40:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 266712064 bytes 2024-09-11T05:40:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 314141696 bytes 2024-09-11T05:40:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 364322816 bytes 2024-09-11T05:40:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 388499456 bytes 2024-09-11T05:40:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 408508416 bytes 2024-09-11T05:40:11.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 500318208 bytes 2024-09-11T05:40:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 553155584 bytes 2024-09-11T05:40:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 553791488 bytes 2024-09-11T05:40:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 588910592 bytes 2024-09-11T05:40:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 602491904 bytes 2024-09-11T05:40:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 706100224 bytes 2024-09-11T05:40:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 710163456 bytes 2024-09-11T05:40:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 733642752 bytes 2024-09-11T05:40:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 751807488 bytes 2024-09-11T05:40:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 827792384 bytes 2024-09-11T05:40:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 850731008 bytes 2024-09-11T05:40:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 870182912 bytes 2024-09-11T05:40:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 896973824 bytes 2024-09-11T05:40:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 937146368 bytes 2024-09-11T05:40:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 973113344 bytes 2024-09-11T05:40:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1020217344 bytes 2024-09-11T05:40:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1035234304 bytes 2024-09-11T05:40:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1057930240 bytes 2024-09-11T05:40:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1090235392 bytes 2024-09-11T05:40:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1138979840 bytes 2024-09-11T05:40:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1196022784 bytes 2024-09-11T05:40:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1266085888 bytes 2024-09-11T05:40:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1301439488 bytes 2024-09-11T05:40:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1379951616 bytes 2024-09-11T05:40:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1397324800 bytes 2024-09-11T05:40:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1416642560 bytes 2024-09-11T05:40:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1457555456 bytes 2024-09-11T05:40:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1493882880 bytes 2024-09-11T05:40:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1539685376 bytes 2024-09-11T05:40:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1583123456 bytes 2024-09-11T05:40:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1619176448 bytes 2024-09-11T05:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1657621504 bytes 2024-09-11T05:40:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1665150976 bytes 2024-09-11T05:40:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1674285056 bytes 2024-09-11T05:40:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1749255168 bytes 2024-09-11T05:40:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1831388160 bytes 2024-09-11T05:40:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1855426560 bytes 2024-09-11T05:40:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1864765440 bytes 2024-09-11T05:40:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1889076224 bytes 2024-09-11T05:40:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1921423360 bytes 2024-09-11T05:40:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1949864960 bytes 2024-09-11T05:40:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1980585984 bytes 2024-09-11T05:40:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2004183040 bytes 2024-09-11T05:40:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2056178688 bytes 2024-09-11T05:40:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2074617856 bytes 2024-09-11T05:40:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2090673152 bytes 2024-09-11T05:40:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2151660544 bytes 2024-09-11T05:40:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2187874304 bytes 2024-09-11T05:40:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2211893248 bytes 2024-09-11T05:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2331737088 bytes 2024-09-11T05:40:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2368983040 bytes 2024-09-11T05:40:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2392509440 bytes 2024-09-11T05:40:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2444222464 bytes 2024-09-11T05:40:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2485985280 bytes 2024-09-11T05:40:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2539863040 bytes 2024-09-11T05:40:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2554917888 bytes 2024-09-11T05:40:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2571424768 bytes 2024-09-11T05:40:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2598636544 bytes 2024-09-11T05:40:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2604132352 bytes 2024-09-11T05:40:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2639645696 bytes 2024-09-11T05:40:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2691299328 bytes 2024-09-11T05:40:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2727179264 bytes 2024-09-11T05:40:19.568 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:40:19.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:40:19.569 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T05:40:19.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:40:19.568+0000 7ff6936a0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 34064) UID: 1000 2024-09-11T05:40:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2751535104 bytes 2024-09-11T05:40:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2828185600 bytes 2024-09-11T05:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2878370816 bytes 2024-09-11T05:40:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 2878965760 bytes 2024-09-11T05:40:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 2950255616 bytes 2024-09-11T05:40:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3012779008 bytes 2024-09-11T05:40:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3037701120 bytes 2024-09-11T05:40:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3063655424 bytes 2024-09-11T05:40:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3202764800 bytes 2024-09-11T05:40:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3239836672 bytes 2024-09-11T05:40:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3253687296 bytes 2024-09-11T05:40:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3329134592 bytes 2024-09-11T05:40:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3361310720 bytes 2024-09-11T05:40:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3370764288 bytes 2024-09-11T05:40:21.570 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:40:21.570 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:40:21.570 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:40:21.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:40:21.571 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:40:21.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:40:21.573 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-11T05:40:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3384654848 bytes 2024-09-11T05:40:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3406978048 bytes 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:40:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:40:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:40:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:40:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-11T05:40:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-11T05:40:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:40:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-11T05:40:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_17': 2024-09-11T05:40:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:40:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:40:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:40:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13441a729d3 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13441a729d3 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:40:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:21 2024 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:21 2024 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:21 2024 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e 2024-09-11T05:40:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:40:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:40:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-11T05:40:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:40:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:40:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:40:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:40:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:40:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 18668544 bytes 2024-09-11T05:40:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 19292160 bytes 2024-09-11T05:40:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 43168768 bytes 2024-09-11T05:40:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 73593856 bytes 2024-09-11T05:40:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 74230784 bytes 2024-09-11T05:40:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 141969408 bytes 2024-09-11T05:40:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 142610432 bytes 2024-09-11T05:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 191905792 bytes 2024-09-11T05:40:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 232249344 bytes 2024-09-11T05:40:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 267429888 bytes 2024-09-11T05:40:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 308698112 bytes 2024-09-11T05:40:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 379578368 bytes 2024-09-11T05:40:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 400716800 bytes 2024-09-11T05:40:24.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 439112704 bytes 2024-09-11T05:40:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 489649152 bytes 2024-09-11T05:40:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 533526528 bytes 2024-09-11T05:40:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 592185344 bytes 2024-09-11T05:40:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 614790144 bytes 2024-09-11T05:40:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 638252032 bytes 2024-09-11T05:40:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 687661056 bytes 2024-09-11T05:40:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 750058496 bytes 2024-09-11T05:40:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 775692288 bytes 2024-09-11T05:40:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 806702080 bytes 2024-09-11T05:40:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 838934528 bytes 2024-09-11T05:40:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 866641920 bytes 2024-09-11T05:40:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 905358336 bytes 2024-09-11T05:40:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 949138432 bytes 2024-09-11T05:40:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 986885120 bytes 2024-09-11T05:40:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1006162944 bytes 2024-09-11T05:40:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1081076736 bytes 2024-09-11T05:40:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1184913408 bytes 2024-09-11T05:40:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1217121280 bytes 2024-09-11T05:40:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1275053056 bytes 2024-09-11T05:40:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1351897088 bytes 2024-09-11T05:40:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1395660800 bytes 2024-09-11T05:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1447087104 bytes 2024-09-11T05:40:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1521895424 bytes 2024-09-11T05:40:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1603675136 bytes 2024-09-11T05:40:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1617927168 bytes 2024-09-11T05:40:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1705012224 bytes 2024-09-11T05:40:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1779390464 bytes 2024-09-11T05:40:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1812550656 bytes 2024-09-11T05:40:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1872210944 bytes 2024-09-11T05:40:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1905835008 bytes 2024-09-11T05:40:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1945653248 bytes 2024-09-11T05:40:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1977932800 bytes 2024-09-11T05:40:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2006492160 bytes 2024-09-11T05:40:29.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2048826368 bytes 2024-09-11T05:40:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2144335872 bytes 2024-09-11T05:40:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2166994944 bytes 2024-09-11T05:40:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2175211520 bytes 2024-09-11T05:40:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2199300096 bytes 2024-09-11T05:40:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2230643712 bytes 2024-09-11T05:40:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2308325376 bytes 2024-09-11T05:40:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2383811584 bytes 2024-09-11T05:40:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2386237440 bytes 2024-09-11T05:40:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2466810880 bytes 2024-09-11T05:40:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2516056064 bytes 2024-09-11T05:40:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2581202944 bytes 2024-09-11T05:40:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2584687616 bytes 2024-09-11T05:40:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2595436544 bytes 2024-09-11T05:40:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2639409152 bytes 2024-09-11T05:40:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2675789824 bytes 2024-09-11T05:40:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2709989376 bytes 2024-09-11T05:40:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:40:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:40:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:40:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-11T05:40:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:40:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-11T05:40:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:40:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_18': 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13526821f7e 2024-09-11T05:40:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13526821f7e 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:31 2024 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:31 2024 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:31 2024 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7f727691-cb67-470b-9ad0-990f37a8f164 2024-09-11T05:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-11T05:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:40:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:40:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:40:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:40:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:40:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 36529152 bytes 2024-09-11T05:40:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 64880640 bytes 2024-09-11T05:40:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 99307520 bytes 2024-09-11T05:40:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 125376512 bytes 2024-09-11T05:40:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 197404672 bytes 2024-09-11T05:40:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 243904512 bytes 2024-09-11T05:40:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 298455040 bytes 2024-09-11T05:40:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 331580416 bytes 2024-09-11T05:40:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 343504896 bytes 2024-09-11T05:40:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 359059456 bytes 2024-09-11T05:40:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 376818688 bytes 2024-09-11T05:40:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 377915392 bytes 2024-09-11T05:40:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 434422784 bytes 2024-09-11T05:40:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 493911040 bytes 2024-09-11T05:40:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 571250688 bytes 2024-09-11T05:40:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 583105536 bytes 2024-09-11T05:40:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 629430272 bytes 2024-09-11T05:40:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 657460224 bytes 2024-09-11T05:40:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 712808448 bytes 2024-09-11T05:40:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 741130240 bytes 2024-09-11T05:40:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 811500544 bytes 2024-09-11T05:40:35.699 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:40:35.696+0000 7f41fb6d6640 -1 reset not still connected to 0x55e796214000 2024-09-11T05:40:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 854052864 bytes 2024-09-11T05:40:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 915020800 bytes 2024-09-11T05:40:36.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 976174080 bytes 2024-09-11T05:40:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1026358272 bytes 2024-09-11T05:40:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1081920512 bytes 2024-09-11T05:40:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1104924672 bytes 2024-09-11T05:40:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1147986944 bytes 2024-09-11T05:40:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1170738176 bytes 2024-09-11T05:40:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1197815808 bytes 2024-09-11T05:40:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1240878080 bytes 2024-09-11T05:40:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1254016000 bytes 2024-09-11T05:40:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1311335424 bytes 2024-09-11T05:40:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1348594688 bytes 2024-09-11T05:40:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1390295040 bytes 2024-09-11T05:40:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1452517376 bytes 2024-09-11T05:40:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1511641088 bytes 2024-09-11T05:40:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1556407296 bytes 2024-09-11T05:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1630265344 bytes 2024-09-11T05:40:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1682269184 bytes 2024-09-11T05:40:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1737049088 bytes 2024-09-11T05:40:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1759575040 bytes 2024-09-11T05:40:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1807152128 bytes 2024-09-11T05:40:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1926466560 bytes 2024-09-11T05:40:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2014150656 bytes 2024-09-11T05:40:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2063841280 bytes 2024-09-11T05:40:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2090226688 bytes 2024-09-11T05:40:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2101818368 bytes 2024-09-11T05:40:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2152155136 bytes 2024-09-11T05:40:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2235005952 bytes 2024-09-11T05:40:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2235635712 bytes 2024-09-11T05:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2344295424 bytes 2024-09-11T05:40:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2447331328 bytes 2024-09-11T05:40:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2479968256 bytes 2024-09-11T05:40:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2507422720 bytes 2024-09-11T05:40:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2553787392 bytes 2024-09-11T05:40:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2584646656 bytes 2024-09-11T05:40:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:40:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:40:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:40:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:40:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:40:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:40:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-11T05:40:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:40:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_19': 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 135e1d6d7449 2024-09-11T05:40:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.135e1d6d7449 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:40 2024 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:40 2024 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:40 2024 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 18a350ae-4c11-4211-aa64-b6afa6f108ba 2024-09-11T05:40:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:40:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:40:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-11T05:40:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:40:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:40:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:40:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:40:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:40:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 23615488 bytes 2024-09-11T05:40:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 65410048 bytes 2024-09-11T05:40:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 69479424 bytes 2024-09-11T05:40:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 118387712 bytes 2024-09-11T05:40:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 145150976 bytes 2024-09-11T05:40:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 164839424 bytes 2024-09-11T05:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 165435392 bytes 2024-09-11T05:40:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 191834112 bytes 2024-09-11T05:40:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 254538752 bytes 2024-09-11T05:40:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 298970112 bytes 2024-09-11T05:40:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 324995072 bytes 2024-09-11T05:40:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 354414592 bytes 2024-09-11T05:40:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 412396544 bytes 2024-09-11T05:40:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 534506496 bytes 2024-09-11T05:40:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 544386048 bytes 2024-09-11T05:40:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 560086016 bytes 2024-09-11T05:40:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 592765952 bytes 2024-09-11T05:40:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 614393856 bytes 2024-09-11T05:40:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 623939584 bytes 2024-09-11T05:40:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 667801600 bytes 2024-09-11T05:40:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 698749952 bytes 2024-09-11T05:40:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 723598336 bytes 2024-09-11T05:40:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 777494528 bytes 2024-09-11T05:40:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 882588672 bytes 2024-09-11T05:40:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 931210240 bytes 2024-09-11T05:40:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 946857984 bytes 2024-09-11T05:40:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 984521728 bytes 2024-09-11T05:40:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 990851072 bytes 2024-09-11T05:40:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1009230848 bytes 2024-09-11T05:40:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1049004032 bytes 2024-09-11T05:40:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1090733056 bytes 2024-09-11T05:40:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1101318144 bytes 2024-09-11T05:40:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1194229760 bytes 2024-09-11T05:40:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1220311040 bytes 2024-09-11T05:40:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1244777472 bytes 2024-09-11T05:40:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1255235584 bytes 2024-09-11T05:40:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1339710464 bytes 2024-09-11T05:40:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1395303424 bytes 2024-09-11T05:40:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1419666432 bytes 2024-09-11T05:40:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1524218880 bytes 2024-09-11T05:40:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1558653952 bytes 2024-09-11T05:40:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1597512704 bytes 2024-09-11T05:40:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1619838976 bytes 2024-09-11T05:40:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1644982272 bytes 2024-09-11T05:40:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1675783168 bytes 2024-09-11T05:40:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1694117888 bytes 2024-09-11T05:40:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1726663680 bytes 2024-09-11T05:40:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1796167680 bytes 2024-09-11T05:40:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1836817408 bytes 2024-09-11T05:40:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1864598528 bytes 2024-09-11T05:40:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1937104896 bytes 2024-09-11T05:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1971360768 bytes 2024-09-11T05:40:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1989725184 bytes 2024-09-11T05:40:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2009080832 bytes 2024-09-11T05:40:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2064142336 bytes 2024-09-11T05:40:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2095671296 bytes 2024-09-11T05:40:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2122805248 bytes 2024-09-11T05:40:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2143849472 bytes 2024-09-11T05:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2226259968 bytes 2024-09-11T05:40:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:40:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:40:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:40:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:40:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2024-09-11T05:40:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-11T05:40:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:40:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:40:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-11T05:40:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:40:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:40:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:40:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-11T05:40:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:40:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:40:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-11T05:40:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:40:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-11T05:40:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_20': 2024-09-11T05:40:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:40:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:40:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:40:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 136a54312803 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.136a54312803 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:49 2024 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:49 2024 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:49 2024 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4607f120-f3a3-4ead-9c81-2256f6efd774 2024-09-11T05:40:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:40:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:40:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-11T05:40:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:40:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:40:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:40:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:40:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:40:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:40:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:40:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 22237184 bytes 2024-09-11T05:40:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 45554688 bytes 2024-09-11T05:40:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 83398656 bytes 2024-09-11T05:40:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 135841792 bytes 2024-09-11T05:40:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 154728448 bytes 2024-09-11T05:40:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 191672320 bytes 2024-09-11T05:40:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 236456960 bytes 2024-09-11T05:40:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 251415552 bytes 2024-09-11T05:40:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 293141504 bytes 2024-09-11T05:40:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 337617920 bytes 2024-09-11T05:40:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 391226368 bytes 2024-09-11T05:40:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 396194816 bytes 2024-09-11T05:40:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 414611456 bytes 2024-09-11T05:40:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 443760640 bytes 2024-09-11T05:40:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 508971008 bytes 2024-09-11T05:40:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 529411072 bytes 2024-09-11T05:40:53.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 545354752 bytes 2024-09-11T05:40:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 563864576 bytes 2024-09-11T05:40:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 620008448 bytes 2024-09-11T05:40:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 649912320 bytes 2024-09-11T05:40:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 666925056 bytes 2024-09-11T05:40:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 713796608 bytes 2024-09-11T05:40:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 745713664 bytes 2024-09-11T05:40:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 805634048 bytes 2024-09-11T05:40:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 870335488 bytes 2024-09-11T05:40:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 913333248 bytes 2024-09-11T05:40:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 952315904 bytes 2024-09-11T05:40:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1021604864 bytes 2024-09-11T05:40:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1045122048 bytes 2024-09-11T05:40:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1049839616 bytes 2024-09-11T05:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1171075072 bytes 2024-09-11T05:40:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1232955392 bytes 2024-09-11T05:40:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1260444672 bytes 2024-09-11T05:40:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1278902272 bytes 2024-09-11T05:40:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1311100928 bytes 2024-09-11T05:40:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1323012096 bytes 2024-09-11T05:40:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1385014272 bytes 2024-09-11T05:40:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1408176128 bytes 2024-09-11T05:40:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1419207680 bytes 2024-09-11T05:40:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1426251776 bytes 2024-09-11T05:40:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1459186688 bytes 2024-09-11T05:40:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1483962368 bytes 2024-09-11T05:40:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1511669760 bytes 2024-09-11T05:40:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1557679104 bytes 2024-09-11T05:40:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1591851008 bytes 2024-09-11T05:40:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1633806336 bytes 2024-09-11T05:40:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1691018240 bytes 2024-09-11T05:40:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1715440640 bytes 2024-09-11T05:40:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1756246016 bytes 2024-09-11T05:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1756850176 bytes 2024-09-11T05:40:57.545 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9751582593133463) > weight (0.25) 2024-09-11T05:40:57.545 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-11T05:40:57.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-11T05:40:57.546 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-11T05:40:57.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T05:40:57.543+0000 7f9bd1e65640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 32298) UID: 1000 2024-09-11T05:40:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1797020672 bytes 2024-09-11T05:40:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1827277824 bytes 2024-09-11T05:40:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1843466240 bytes 2024-09-11T05:40:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1901012992 bytes 2024-09-11T05:40:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1975702528 bytes 2024-09-11T05:40:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 1990407168 bytes 2024-09-11T05:40:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2068600832 bytes 2024-09-11T05:40:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2110322688 bytes 2024-09-11T05:40:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2129733632 bytes 2024-09-11T05:40:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2171609088 bytes 2024-09-11T05:40:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2182418432 bytes 2024-09-11T05:40:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2203129856 bytes 2024-09-11T05:40:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2296419328 bytes 2024-09-11T05:40:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2403034112 bytes 2024-09-11T05:40:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2486071296 bytes 2024-09-11T05:40:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2551633920 bytes 2024-09-11T05:41:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2577930240 bytes 2024-09-11T05:41:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2638374912 bytes 2024-09-11T05:41:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2692334592 bytes 2024-09-11T05:41:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2739076096 bytes 2024-09-11T05:41:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2757733376 bytes 2024-09-11T05:41:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2772083712 bytes 2024-09-11T05:41:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2774066176 bytes 2024-09-11T05:41:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2805551104 bytes 2024-09-11T05:41:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2937763840 bytes 2024-09-11T05:41:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3002218496 bytes 2024-09-11T05:41:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3009935360 bytes 2024-09-11T05:41:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3041142784 bytes 2024-09-11T05:41:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3112292352 bytes 2024-09-11T05:41:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3192988672 bytes 2024-09-11T05:41:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3221673984 bytes 2024-09-11T05:41:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3261223936 bytes 2024-09-11T05:41:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3267361792 bytes 2024-09-11T05:41:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3355759616 bytes 2024-09-11T05:41:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3398462464 bytes 2024-09-11T05:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3478232064 bytes 2024-09-11T05:41:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3518205952 bytes 2024-09-11T05:41:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3556067328 bytes 2024-09-11T05:41:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3569336320 bytes 2024-09-11T05:41:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3615679488 bytes 2024-09-11T05:41:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3620999168 bytes 2024-09-11T05:41:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3695315968 bytes 2024-09-11T05:41:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3729704960 bytes 2024-09-11T05:41:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:41:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-11T05:41:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:41:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-11T05:41:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:41:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_21': 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13766d5bed08 2024-09-11T05:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13766d5bed08 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:03 2024 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:03 2024 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:03 2024 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d6371132-753a-4a4f-8524-8425689888b8 2024-09-11T05:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:41:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:41:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-11T05:41:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:41:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2024-09-11T05:41:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:41:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:41:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 42576896 bytes 2024-09-11T05:41:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 158445568 bytes 2024-09-11T05:41:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 230519808 bytes 2024-09-11T05:41:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 282812416 bytes 2024-09-11T05:41:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 323481600 bytes 2024-09-11T05:41:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 366789632 bytes 2024-09-11T05:41:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 367403008 bytes 2024-09-11T05:41:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 372286464 bytes 2024-09-11T05:41:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 405679104 bytes 2024-09-11T05:41:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 461287424 bytes 2024-09-11T05:41:06.547 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-11T05:41:06.547 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:41:06.547 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-11T05:41:06.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T05:41:06.548 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T05:41:06.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T05:41:06.551 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-11T05:41:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 501539840 bytes 2024-09-11T05:41:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 614821888 bytes 2024-09-11T05:41:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 672344064 bytes 2024-09-11T05:41:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 707723264 bytes 2024-09-11T05:41:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 753582080 bytes 2024-09-11T05:41:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 781845504 bytes 2024-09-11T05:41:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 808328192 bytes 2024-09-11T05:41:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 835473408 bytes 2024-09-11T05:41:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 856012800 bytes 2024-09-11T05:41:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 928106496 bytes 2024-09-11T05:41:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1039440896 bytes 2024-09-11T05:41:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1049239552 bytes 2024-09-11T05:41:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1051402240 bytes 2024-09-11T05:41:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1078605824 bytes 2024-09-11T05:41:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1167069184 bytes 2024-09-11T05:41:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1216071680 bytes 2024-09-11T05:41:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1255444480 bytes 2024-09-11T05:41:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1299174400 bytes 2024-09-11T05:41:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1326869504 bytes 2024-09-11T05:41:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1340593152 bytes 2024-09-11T05:41:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1358443520 bytes 2024-09-11T05:41:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1384890368 bytes 2024-09-11T05:41:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1475266560 bytes 2024-09-11T05:41:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1546182656 bytes 2024-09-11T05:41:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1569316864 bytes 2024-09-11T05:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1591532544 bytes 2024-09-11T05:41:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1681384448 bytes 2024-09-11T05:41:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1805841408 bytes 2024-09-11T05:41:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1887181824 bytes 2024-09-11T05:41:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1909705728 bytes 2024-09-11T05:41:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1923820544 bytes 2024-09-11T05:41:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1937625088 bytes 2024-09-11T05:41:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1940424704 bytes 2024-09-11T05:41:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1986932736 bytes 2024-09-11T05:41:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2027983872 bytes 2024-09-11T05:41:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2060912640 bytes 2024-09-11T05:41:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2081358848 bytes 2024-09-11T05:41:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2161836032 bytes 2024-09-11T05:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2242061312 bytes 2024-09-11T05:41:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2297740288 bytes 2024-09-11T05:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2309227520 bytes 2024-09-11T05:41:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2380465152 bytes 2024-09-11T05:41:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2408474624 bytes 2024-09-11T05:41:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2470662144 bytes 2024-09-11T05:41:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2497131520 bytes 2024-09-11T05:41:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2551231488 bytes 2024-09-11T05:41:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2577168384 bytes 2024-09-11T05:41:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2596513792 bytes 2024-09-11T05:41:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2633411584 bytes 2024-09-11T05:41:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2687903744 bytes 2024-09-11T05:41:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2701325312 bytes 2024-09-11T05:41:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2756745216 bytes 2024-09-11T05:41:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2785623040 bytes 2024-09-11T05:41:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2837899264 bytes 2024-09-11T05:41:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2852851712 bytes 2024-09-11T05:41:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2868699136 bytes 2024-09-11T05:41:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2922944512 bytes 2024-09-11T05:41:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2952354816 bytes 2024-09-11T05:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2977084416 bytes 2024-09-11T05:41:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3004798976 bytes 2024-09-11T05:41:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:41:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-11T05:41:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:41:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-11T05:41:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:41:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-11T05:41:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_22': 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 138798543c63 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.138798543c63 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:14 2024 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:14 2024 2024-09-11T05:41:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:14 2024 2024-09-11T05:41:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:41:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:41:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: db729295-01a4-4d5f-aec1-17d8b32aefee 2024-09-11T05:41:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:41:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:41:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:41:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:41:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:41:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 49308672 bytes 2024-09-11T05:41:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 144013312 bytes 2024-09-11T05:41:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 162072576 bytes 2024-09-11T05:41:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 184890368 bytes 2024-09-11T05:41:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 255508480 bytes 2024-09-11T05:41:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 256111616 bytes 2024-09-11T05:41:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 318157824 bytes 2024-09-11T05:41:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 370177024 bytes 2024-09-11T05:41:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 449477632 bytes 2024-09-11T05:41:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 507083776 bytes 2024-09-11T05:41:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 553974784 bytes 2024-09-11T05:41:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 634545152 bytes 2024-09-11T05:41:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 654862336 bytes 2024-09-11T05:41:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 747404288 bytes 2024-09-11T05:41:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 774448128 bytes 2024-09-11T05:41:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 804242432 bytes 2024-09-11T05:41:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 844388352 bytes 2024-09-11T05:41:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 891682816 bytes 2024-09-11T05:41:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 916271104 bytes 2024-09-11T05:41:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 979321856 bytes 2024-09-11T05:41:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1005180928 bytes 2024-09-11T05:41:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1021069312 bytes 2024-09-11T05:41:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1064403968 bytes 2024-09-11T05:41:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1089359872 bytes 2024-09-11T05:41:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1103107072 bytes 2024-09-11T05:41:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1142381568 bytes 2024-09-11T05:41:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1169187840 bytes 2024-09-11T05:41:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1226877952 bytes 2024-09-11T05:41:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1227488256 bytes 2024-09-11T05:41:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1248634880 bytes 2024-09-11T05:41:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1313650688 bytes 2024-09-11T05:41:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1326841856 bytes 2024-09-11T05:41:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1398354944 bytes 2024-09-11T05:41:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1439637504 bytes 2024-09-11T05:41:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1474932736 bytes 2024-09-11T05:41:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1516027904 bytes 2024-09-11T05:41:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1548435456 bytes 2024-09-11T05:41:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1591264256 bytes 2024-09-11T05:41:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1655945216 bytes 2024-09-11T05:41:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1750932480 bytes 2024-09-11T05:41:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1769920512 bytes 2024-09-11T05:41:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1809240064 bytes 2024-09-11T05:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1819318272 bytes 2024-09-11T05:41:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1844590592 bytes 2024-09-11T05:41:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1957192704 bytes 2024-09-11T05:41:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1959999488 bytes 2024-09-11T05:41:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2027792384 bytes 2024-09-11T05:41:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2038962176 bytes 2024-09-11T05:41:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2104873984 bytes 2024-09-11T05:41:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2106397696 bytes 2024-09-11T05:41:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2171268096 bytes 2024-09-11T05:41:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2211450880 bytes 2024-09-11T05:41:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2240310272 bytes 2024-09-11T05:41:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2259267584 bytes 2024-09-11T05:41:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2289236992 bytes 2024-09-11T05:41:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2314393600 bytes 2024-09-11T05:41:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2328931328 bytes 2024-09-11T05:41:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2329570304 bytes 2024-09-11T05:41:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2382019584 bytes 2024-09-11T05:41:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2416137216 bytes 2024-09-11T05:41:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2461501440 bytes 2024-09-11T05:41:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2468783104 bytes 2024-09-11T05:41:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2524685312 bytes 2024-09-11T05:41:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2606342144 bytes 2024-09-11T05:41:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2672686080 bytes 2024-09-11T05:41:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2749697024 bytes 2024-09-11T05:41:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2782072832 bytes 2024-09-11T05:41:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2797115392 bytes 2024-09-11T05:41:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2822421504 bytes 2024-09-11T05:41:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2870435840 bytes 2024-09-11T05:41:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2900347904 bytes 2024-09-11T05:41:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2972537856 bytes 2024-09-11T05:41:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3004583936 bytes 2024-09-11T05:41:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3015974912 bytes 2024-09-11T05:41:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3068331008 bytes 2024-09-11T05:41:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3103291392 bytes 2024-09-11T05:41:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3113076736 bytes 2024-09-11T05:41:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3153970176 bytes 2024-09-11T05:41:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3154603008 bytes 2024-09-11T05:41:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3179916288 bytes 2024-09-11T05:41:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3226595328 bytes 2024-09-11T05:41:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3235620864 bytes 2024-09-11T05:41:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3298325504 bytes 2024-09-11T05:41:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3376347136 bytes 2024-09-11T05:41:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3404771328 bytes 2024-09-11T05:41:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3461784576 bytes 2024-09-11T05:41:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3504858112 bytes 2024-09-11T05:41:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3536652288 bytes 2024-09-11T05:41:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3605046272 bytes 2024-09-11T05:41:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3615606784 bytes 2024-09-11T05:41:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:41:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:41:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-11T05:41:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:41:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:41:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-11T05:41:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:41:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-11T05:41:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_23': 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 138f6efc21c7 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.138f6efc21c7 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:28 2024 2024-09-11T05:41:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:28 2024 2024-09-11T05:41:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:28 2024 2024-09-11T05:41:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:41:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:41:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7 2024-09-11T05:41:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:41:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:41:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-11T05:41:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:41:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:41:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2024-09-11T05:41:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:41:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:41:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 32230400 bytes 2024-09-11T05:41:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 113576960 bytes 2024-09-11T05:41:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 134908928 bytes 2024-09-11T05:41:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 166159360 bytes 2024-09-11T05:41:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 197364736 bytes 2024-09-11T05:41:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 230064128 bytes 2024-09-11T05:41:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 263411712 bytes 2024-09-11T05:41:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 344311808 bytes 2024-09-11T05:41:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 384939008 bytes 2024-09-11T05:41:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 459069440 bytes 2024-09-11T05:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 528385024 bytes 2024-09-11T05:41:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 546859008 bytes 2024-09-11T05:41:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 594914304 bytes 2024-09-11T05:41:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 629669888 bytes 2024-09-11T05:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 639601664 bytes 2024-09-11T05:41:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 654158848 bytes 2024-09-11T05:41:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 676985856 bytes 2024-09-11T05:41:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 712445952 bytes 2024-09-11T05:41:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 758852608 bytes 2024-09-11T05:41:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 832669696 bytes 2024-09-11T05:41:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 880471040 bytes 2024-09-11T05:41:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 933287936 bytes 2024-09-11T05:41:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 946236416 bytes 2024-09-11T05:41:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 970530816 bytes 2024-09-11T05:41:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1050873856 bytes 2024-09-11T05:41:33.574 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5187299980292489) > weight (0.25) 2024-09-11T05:41:33.574 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T05:41:33.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T05:41:33.575 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T05:41:33.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:41:33.575+0000 7fde30ee3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 28925) UID: 1000 2024-09-11T05:41:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1111537664 bytes 2024-09-11T05:41:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1122320384 bytes 2024-09-11T05:41:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1122931712 bytes 2024-09-11T05:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1135947776 bytes 2024-09-11T05:41:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1162348544 bytes 2024-09-11T05:41:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1276654592 bytes 2024-09-11T05:41:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1315631104 bytes 2024-09-11T05:41:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1370876928 bytes 2024-09-11T05:41:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1412022272 bytes 2024-09-11T05:41:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1429850112 bytes 2024-09-11T05:41:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1445478400 bytes 2024-09-11T05:41:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1543840768 bytes 2024-09-11T05:41:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1581719552 bytes 2024-09-11T05:41:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1622636544 bytes 2024-09-11T05:41:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1649818624 bytes 2024-09-11T05:41:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1701015552 bytes 2024-09-11T05:41:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1784132608 bytes 2024-09-11T05:41:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1801161728 bytes 2024-09-11T05:41:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1801798656 bytes 2024-09-11T05:41:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1845638144 bytes 2024-09-11T05:41:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1847028736 bytes 2024-09-11T05:41:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1914212352 bytes 2024-09-11T05:41:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1951631360 bytes 2024-09-11T05:41:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1977053184 bytes 2024-09-11T05:41:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2031749120 bytes 2024-09-11T05:41:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2052818944 bytes 2024-09-11T05:41:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2072649728 bytes 2024-09-11T05:41:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2151826432 bytes 2024-09-11T05:41:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2186557440 bytes 2024-09-11T05:41:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2256921600 bytes 2024-09-11T05:41:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2312573952 bytes 2024-09-11T05:41:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2374031360 bytes 2024-09-11T05:41:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2399049728 bytes 2024-09-11T05:41:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2405554176 bytes 2024-09-11T05:41:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2432501760 bytes 2024-09-11T05:41:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2513972224 bytes 2024-09-11T05:41:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2634606592 bytes 2024-09-11T05:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2641063936 bytes 2024-09-11T05:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2677479424 bytes 2024-09-11T05:41:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2691757056 bytes 2024-09-11T05:41:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2727813120 bytes 2024-09-11T05:41:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2734050304 bytes 2024-09-11T05:41:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2757755904 bytes 2024-09-11T05:41:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2817274880 bytes 2024-09-11T05:41:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2863288320 bytes 2024-09-11T05:41:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:41:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:41:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:41:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-11T05:41:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:41:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:41:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-11T05:41:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:41:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-11T05:41:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:41:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_24': 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 139720090e6c 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.139720090e6c 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:41:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:40 2024 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:40 2024 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:40 2024 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 73f51a71-2b0c-4c23-b0d3-ddf1f35a1a67 2024-09-11T05:41:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:41:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:41:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:41:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:41:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:41:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 27527168 bytes 2024-09-11T05:41:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 40494080 bytes 2024-09-11T05:41:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 139138048 bytes 2024-09-11T05:41:41.576 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T05:41:41.576 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:41:41.577 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T05:41:41.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T05:41:41.577 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T05:41:41.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T05:41:41.582 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-11T05:41:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 244803584 bytes 2024-09-11T05:41:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 297054208 bytes 2024-09-11T05:41:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 372239360 bytes 2024-09-11T05:41:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 397852672 bytes 2024-09-11T05:41:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 421779456 bytes 2024-09-11T05:41:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 464121856 bytes 2024-09-11T05:41:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 510835712 bytes 2024-09-11T05:41:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 546607104 bytes 2024-09-11T05:41:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 592979968 bytes 2024-09-11T05:41:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 598786048 bytes 2024-09-11T05:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 638194688 bytes 2024-09-11T05:41:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 707118080 bytes 2024-09-11T05:41:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 716050432 bytes 2024-09-11T05:41:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 759153664 bytes 2024-09-11T05:41:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 827121664 bytes 2024-09-11T05:41:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 850025472 bytes 2024-09-11T05:41:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 887986176 bytes 2024-09-11T05:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 976123904 bytes 2024-09-11T05:41:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 985226240 bytes 2024-09-11T05:41:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 985815040 bytes 2024-09-11T05:41:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1039721472 bytes 2024-09-11T05:41:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1069448192 bytes 2024-09-11T05:41:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1081662464 bytes 2024-09-11T05:41:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1123456000 bytes 2024-09-11T05:41:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1187479552 bytes 2024-09-11T05:41:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1233923072 bytes 2024-09-11T05:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1291908096 bytes 2024-09-11T05:41:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1385586688 bytes 2024-09-11T05:41:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1417630720 bytes 2024-09-11T05:41:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1439817728 bytes 2024-09-11T05:41:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1487186944 bytes 2024-09-11T05:41:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1543899136 bytes 2024-09-11T05:41:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1565938688 bytes 2024-09-11T05:41:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1601940480 bytes 2024-09-11T05:41:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1626113024 bytes 2024-09-11T05:41:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1659140096 bytes 2024-09-11T05:41:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1742393344 bytes 2024-09-11T05:41:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1761225728 bytes 2024-09-11T05:41:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1778407424 bytes 2024-09-11T05:41:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1808005120 bytes 2024-09-11T05:41:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1848376320 bytes 2024-09-11T05:41:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1877643264 bytes 2024-09-11T05:41:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1950381056 bytes 2024-09-11T05:41:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1989854208 bytes 2024-09-11T05:41:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2014949376 bytes 2024-09-11T05:41:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2045761536 bytes 2024-09-11T05:41:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2092685312 bytes 2024-09-11T05:41:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2097921024 bytes 2024-09-11T05:41:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2116323328 bytes 2024-09-11T05:41:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2158721024 bytes 2024-09-11T05:41:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2189591552 bytes 2024-09-11T05:41:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2245273600 bytes 2024-09-11T05:41:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2264580096 bytes 2024-09-11T05:41:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2323787776 bytes 2024-09-11T05:41:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2394981376 bytes 2024-09-11T05:41:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2429799424 bytes 2024-09-11T05:41:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2439660544 bytes 2024-09-11T05:41:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2440276992 bytes 2024-09-11T05:41:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2543912960 bytes 2024-09-11T05:41:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2562939904 bytes 2024-09-11T05:41:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2617451520 bytes 2024-09-11T05:41:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2648376320 bytes 2024-09-11T05:41:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2695028736 bytes 2024-09-11T05:41:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2776258560 bytes 2024-09-11T05:41:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2824002560 bytes 2024-09-11T05:41:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2848476160 bytes 2024-09-11T05:41:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2874892288 bytes 2024-09-11T05:41:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2951531520 bytes 2024-09-11T05:41:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3019598848 bytes 2024-09-11T05:41:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3051146240 bytes 2024-09-11T05:41:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3100685312 bytes 2024-09-11T05:41:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3135351808 bytes 2024-09-11T05:41:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3159147520 bytes 2024-09-11T05:41:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3228231680 bytes 2024-09-11T05:41:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3241092096 bytes 2024-09-11T05:41:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3320455168 bytes 2024-09-11T05:41:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3373876224 bytes 2024-09-11T05:41:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3489059840 bytes 2024-09-11T05:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3537490944 bytes 2024-09-11T05:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3602233344 bytes 2024-09-11T05:41:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3653404672 bytes 2024-09-11T05:41:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3713396736 bytes 2024-09-11T05:41:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3729446912 bytes 2024-09-11T05:41:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3770219520 bytes 2024-09-11T05:41:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3831577600 bytes 2024-09-11T05:41:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3839027200 bytes 2024-09-11T05:41:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3855993856 bytes 2024-09-11T05:41:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3890694144 bytes 2024-09-11T05:41:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3921245184 bytes 2024-09-11T05:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3958108160 bytes 2024-09-11T05:41:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3998348288 bytes 2024-09-11T05:41:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4051378176 bytes 2024-09-11T05:41:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4077594624 bytes 2024-09-11T05:41:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:41:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:41:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:41:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:41:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-11T05:41:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:41:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:41:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:41:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-11T05:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:41:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-11T05:41:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:41:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-11T05:41:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_25': 2024-09-11T05:41:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:41:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13aa8a79af70 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13aa8a79af70 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:55 2024 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:55 2024 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:55 2024 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:41:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a87873a5-1cad-45f5-a638-56e77deab3e3 2024-09-11T05:41:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:41:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:41:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:41:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:41:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:41:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 18176000 bytes 2024-09-11T05:41:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 40337408 bytes 2024-09-11T05:41:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 99148800 bytes 2024-09-11T05:41:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 125165568 bytes 2024-09-11T05:41:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 126741504 bytes 2024-09-11T05:41:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 222164992 bytes 2024-09-11T05:41:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 268181504 bytes 2024-09-11T05:41:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 296973312 bytes 2024-09-11T05:41:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 323383296 bytes 2024-09-11T05:41:57.086 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:41:57.083+0000 7f41fb6d6640 -1 reset not still connected to 0x55e79b96f6c0 2024-09-11T05:41:57.086 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:41:57.083+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7a31af860 2024-09-11T05:41:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 324396032 bytes 2024-09-11T05:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 374765568 bytes 2024-09-11T05:41:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 418265088 bytes 2024-09-11T05:41:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 484531200 bytes 2024-09-11T05:41:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 514616320 bytes 2024-09-11T05:41:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 555302912 bytes 2024-09-11T05:41:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 575511552 bytes 2024-09-11T05:41:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 595740672 bytes 2024-09-11T05:41:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 627417088 bytes 2024-09-11T05:41:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 675134464 bytes 2024-09-11T05:41:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 714987520 bytes 2024-09-11T05:41:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 791294976 bytes 2024-09-11T05:41:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 825067520 bytes 2024-09-11T05:41:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 861470720 bytes 2024-09-11T05:41:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 885896192 bytes 2024-09-11T05:41:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 916019200 bytes 2024-09-11T05:41:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1003492352 bytes 2024-09-11T05:41:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1023272960 bytes 2024-09-11T05:41:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1125209088 bytes 2024-09-11T05:41:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1169538048 bytes 2024-09-11T05:41:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1184659456 bytes 2024-09-11T05:42:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1192386560 bytes 2024-09-11T05:42:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1242668032 bytes 2024-09-11T05:42:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1313112064 bytes 2024-09-11T05:42:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1355357184 bytes 2024-09-11T05:42:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1361890304 bytes 2024-09-11T05:42:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1478103040 bytes 2024-09-11T05:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1526129664 bytes 2024-09-11T05:42:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1554949120 bytes 2024-09-11T05:42:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1589853184 bytes 2024-09-11T05:42:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1625760768 bytes 2024-09-11T05:42:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1676257280 bytes 2024-09-11T05:42:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-11T05:42:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-11T05:42:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-11T05:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_26': 2024-09-11T05:42:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13b37a114873 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13b37a114873 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:01 2024 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:01 2024 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:01 2024 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 95e24eb9-a635-4d8b-b286-5bc01f045ba1 2024-09-11T05:42:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:42:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2024-09-11T05:42:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 38810624 bytes 2024-09-11T05:42:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 83336192 bytes 2024-09-11T05:42:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 130781184 bytes 2024-09-11T05:42:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 208988160 bytes 2024-09-11T05:42:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 277709824 bytes 2024-09-11T05:42:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 287647744 bytes 2024-09-11T05:42:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 318746624 bytes 2024-09-11T05:42:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 345965568 bytes 2024-09-11T05:42:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 397394944 bytes 2024-09-11T05:42:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 471303168 bytes 2024-09-11T05:42:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 495308800 bytes 2024-09-11T05:42:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 513121280 bytes 2024-09-11T05:42:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 606222336 bytes 2024-09-11T05:42:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 632990720 bytes 2024-09-11T05:42:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 672186368 bytes 2024-09-11T05:42:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 710431744 bytes 2024-09-11T05:42:05.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 784898048 bytes 2024-09-11T05:42:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 837226496 bytes 2024-09-11T05:42:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 844390400 bytes 2024-09-11T05:42:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 887291904 bytes 2024-09-11T05:42:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 923781120 bytes 2024-09-11T05:42:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 967040000 bytes 2024-09-11T05:42:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1007365120 bytes 2024-09-11T05:42:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1049307136 bytes 2024-09-11T05:42:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1094529024 bytes 2024-09-11T05:42:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1101250560 bytes 2024-09-11T05:42:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1101856768 bytes 2024-09-11T05:42:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1129326592 bytes 2024-09-11T05:42:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1201277952 bytes 2024-09-11T05:42:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1221110784 bytes 2024-09-11T05:42:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1263570944 bytes 2024-09-11T05:42:07.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1314357248 bytes 2024-09-11T05:42:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1320980480 bytes 2024-09-11T05:42:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1398773760 bytes 2024-09-11T05:42:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1437083648 bytes 2024-09-11T05:42:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1490498560 bytes 2024-09-11T05:42:08.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1499541504 bytes 2024-09-11T05:42:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1522135040 bytes 2024-09-11T05:42:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1622887424 bytes 2024-09-11T05:42:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1694739456 bytes 2024-09-11T05:42:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1695349760 bytes 2024-09-11T05:42:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1734635520 bytes 2024-09-11T05:42:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1735243776 bytes 2024-09-11T05:42:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1747641344 bytes 2024-09-11T05:42:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1783468032 bytes 2024-09-11T05:42:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1821582336 bytes 2024-09-11T05:42:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1946819584 bytes 2024-09-11T05:42:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1986329600 bytes 2024-09-11T05:42:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2041533440 bytes 2024-09-11T05:42:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2120324096 bytes 2024-09-11T05:42:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2137786368 bytes 2024-09-11T05:42:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2204831744 bytes 2024-09-11T05:42:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2230080512 bytes 2024-09-11T05:42:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2259821568 bytes 2024-09-11T05:42:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2315381760 bytes 2024-09-11T05:42:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2363062272 bytes 2024-09-11T05:42:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2422067200 bytes 2024-09-11T05:42:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2476385280 bytes 2024-09-11T05:42:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2497731584 bytes 2024-09-11T05:42:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2526191616 bytes 2024-09-11T05:42:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2582243328 bytes 2024-09-11T05:42:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2608835584 bytes 2024-09-11T05:42:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2616916992 bytes 2024-09-11T05:42:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2671337472 bytes 2024-09-11T05:42:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2712212480 bytes 2024-09-11T05:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2763282432 bytes 2024-09-11T05:42:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2856885248 bytes 2024-09-11T05:42:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2922120192 bytes 2024-09-11T05:42:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3004527616 bytes 2024-09-11T05:42:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3023828992 bytes 2024-09-11T05:42:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3056325632 bytes 2024-09-11T05:42:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3077778432 bytes 2024-09-11T05:42:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3171832832 bytes 2024-09-11T05:42:13.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3243921408 bytes 2024-09-11T05:42:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3310502912 bytes 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2024-09-11T05:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-11T05:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:42:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-11T05:42:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-11T05:42:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_27': 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13bc7b256089 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13bc7b256089 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:13 2024 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:13 2024 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:13 2024 2024-09-11T05:42:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1641c3d7-e0a2-407c-a874-91a1ef797a0c 2024-09-11T05:42:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-11T05:42:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:42:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:42:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:15.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 53101568 bytes 2024-09-11T05:42:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 111275008 bytes 2024-09-11T05:42:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 129588224 bytes 2024-09-11T05:42:15.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 130210816 bytes 2024-09-11T05:42:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 192594944 bytes 2024-09-11T05:42:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 229209088 bytes 2024-09-11T05:42:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 283581440 bytes 2024-09-11T05:42:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 328609792 bytes 2024-09-11T05:42:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 403662848 bytes 2024-09-11T05:42:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 406082560 bytes 2024-09-11T05:42:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 441046016 bytes 2024-09-11T05:42:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 460692480 bytes 2024-09-11T05:42:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 504570880 bytes 2024-09-11T05:42:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 572871680 bytes 2024-09-11T05:42:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 623980544 bytes 2024-09-11T05:42:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 663918592 bytes 2024-09-11T05:42:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 686319616 bytes 2024-09-11T05:42:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 747753472 bytes 2024-09-11T05:42:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 828026880 bytes 2024-09-11T05:42:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 846956544 bytes 2024-09-11T05:42:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 914284544 bytes 2024-09-11T05:42:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 950374400 bytes 2024-09-11T05:42:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 976964608 bytes 2024-09-11T05:42:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1030889472 bytes 2024-09-11T05:42:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1104450560 bytes 2024-09-11T05:42:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1183309824 bytes 2024-09-11T05:42:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1222427648 bytes 2024-09-11T05:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1247847424 bytes 2024-09-11T05:42:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1317734400 bytes 2024-09-11T05:42:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1397828608 bytes 2024-09-11T05:42:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1432890368 bytes 2024-09-11T05:42:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1515554816 bytes 2024-09-11T05:42:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1547670528 bytes 2024-09-11T05:42:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1570277376 bytes 2024-09-11T05:42:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1632403456 bytes 2024-09-11T05:42:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1744470016 bytes 2024-09-11T05:42:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1777827840 bytes 2024-09-11T05:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1822454784 bytes 2024-09-11T05:42:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1844592640 bytes 2024-09-11T05:42:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1886601216 bytes 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-11T05:42:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:42:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-11T05:42:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-11T05:42:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_28': 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13c56e062586 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13c56e062586 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:20 2024 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:20 2024 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:20 2024 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 240aff3e-f4c4-4e78-89e9-f002bb8ec31b 2024-09-11T05:42:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:42:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 48471040 bytes 2024-09-11T05:42:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 51266560 bytes 2024-09-11T05:42:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 122304512 bytes 2024-09-11T05:42:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 157151232 bytes 2024-09-11T05:42:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 185189376 bytes 2024-09-11T05:42:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 217776128 bytes 2024-09-11T05:42:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 231980032 bytes 2024-09-11T05:42:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 285745152 bytes 2024-09-11T05:42:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 428561408 bytes 2024-09-11T05:42:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 470890496 bytes 2024-09-11T05:42:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 508272640 bytes 2024-09-11T05:42:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 539811840 bytes 2024-09-11T05:42:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 620895232 bytes 2024-09-11T05:42:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 629568512 bytes 2024-09-11T05:42:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 679396352 bytes 2024-09-11T05:42:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 689242112 bytes 2024-09-11T05:42:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 708789248 bytes 2024-09-11T05:42:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 769916928 bytes 2024-09-11T05:42:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 796441600 bytes 2024-09-11T05:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 840518656 bytes 2024-09-11T05:42:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 904177664 bytes 2024-09-11T05:42:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 998020096 bytes 2024-09-11T05:42:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1123016704 bytes 2024-09-11T05:42:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1142263808 bytes 2024-09-11T05:42:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1155290112 bytes 2024-09-11T05:42:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1199148032 bytes 2024-09-11T05:42:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1281207296 bytes 2024-09-11T05:42:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1336615936 bytes 2024-09-11T05:42:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1338734592 bytes 2024-09-11T05:42:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1371251712 bytes 2024-09-11T05:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1440125952 bytes 2024-09-11T05:42:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1482613760 bytes 2024-09-11T05:42:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1533094912 bytes 2024-09-11T05:42:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1540725760 bytes 2024-09-11T05:42:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1589832704 bytes 2024-09-11T05:42:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1619929088 bytes 2024-09-11T05:42:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1658923008 bytes 2024-09-11T05:42:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1692351488 bytes 2024-09-11T05:42:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1702154240 bytes 2024-09-11T05:42:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1729292288 bytes 2024-09-11T05:42:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1839583232 bytes 2024-09-11T05:42:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1889211392 bytes 2024-09-11T05:42:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1904210944 bytes 2024-09-11T05:42:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1976406016 bytes 2024-09-11T05:42:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2012358656 bytes 2024-09-11T05:42:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2070778880 bytes 2024-09-11T05:42:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2073079808 bytes 2024-09-11T05:42:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2125365248 bytes 2024-09-11T05:42:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2139777024 bytes 2024-09-11T05:42:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2197837824 bytes 2024-09-11T05:42:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2240696320 bytes 2024-09-11T05:42:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2245413888 bytes 2024-09-11T05:42:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2313202688 bytes 2024-09-11T05:42:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2313897984 bytes 2024-09-11T05:42:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-11T05:42:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-11T05:42:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-11T05:42:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_29': 2024-09-11T05:42:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13cf95313932 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13cf95313932 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:29 2024 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:29 2024 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:29 2024 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2c6de312-17ab-498c-82cf-a905326cbb84 2024-09-11T05:42:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-11T05:42:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:42:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:42:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 44262400 bytes 2024-09-11T05:42:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 61760512 bytes 2024-09-11T05:42:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 102720512 bytes 2024-09-11T05:42:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 160549888 bytes 2024-09-11T05:42:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 191291392 bytes 2024-09-11T05:42:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 211536896 bytes 2024-09-11T05:42:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 276470784 bytes 2024-09-11T05:42:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 310162432 bytes 2024-09-11T05:42:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 345654272 bytes 2024-09-11T05:42:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 360138752 bytes 2024-09-11T05:42:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 411493376 bytes 2024-09-11T05:42:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 448051200 bytes 2024-09-11T05:42:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 467867648 bytes 2024-09-11T05:42:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 564250624 bytes 2024-09-11T05:42:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 584914944 bytes 2024-09-11T05:42:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 640278528 bytes 2024-09-11T05:42:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 698656768 bytes 2024-09-11T05:42:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 720217088 bytes 2024-09-11T05:42:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 826207232 bytes 2024-09-11T05:42:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 883461120 bytes 2024-09-11T05:42:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 961303552 bytes 2024-09-11T05:42:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 994418688 bytes 2024-09-11T05:42:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1037767680 bytes 2024-09-11T05:42:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1063626752 bytes 2024-09-11T05:42:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1156149248 bytes 2024-09-11T05:42:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1185954816 bytes 2024-09-11T05:42:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1206707200 bytes 2024-09-11T05:42:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1303910400 bytes 2024-09-11T05:42:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1335142400 bytes 2024-09-11T05:42:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1368444928 bytes 2024-09-11T05:42:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1385824256 bytes 2024-09-11T05:42:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1439309824 bytes 2024-09-11T05:42:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1444034560 bytes 2024-09-11T05:42:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1466672128 bytes 2024-09-11T05:42:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1502659584 bytes 2024-09-11T05:42:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1541723136 bytes 2024-09-11T05:42:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1567212544 bytes 2024-09-11T05:42:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1608913920 bytes 2024-09-11T05:42:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1626246144 bytes 2024-09-11T05:42:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1710822400 bytes 2024-09-11T05:42:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1771522048 bytes 2024-09-11T05:42:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1852913664 bytes 2024-09-11T05:42:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1905846272 bytes 2024-09-11T05:42:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1978717184 bytes 2024-09-11T05:42:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2026796032 bytes 2024-09-11T05:42:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2069749760 bytes 2024-09-11T05:42:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2128020480 bytes 2024-09-11T05:42:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2146907136 bytes 2024-09-11T05:42:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:42:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-11T05:42:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-11T05:42:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_30': 2024-09-11T05:42:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13d9c90f0861 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13d9c90f0861 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:37 2024 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:37 2024 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:37 2024 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1c1a7c61-6e2b-48cc-80d9-2dcd7ffe860a 2024-09-11T05:42:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:38.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-11T05:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:39.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 3076096 bytes 2024-09-11T05:42:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 40741888 bytes 2024-09-11T05:42:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 139766784 bytes 2024-09-11T05:42:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 172188672 bytes 2024-09-11T05:42:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 228845568 bytes 2024-09-11T05:42:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 291425280 bytes 2024-09-11T05:42:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 308030464 bytes 2024-09-11T05:42:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 374105088 bytes 2024-09-11T05:42:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 395122688 bytes 2024-09-11T05:42:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 413395968 bytes 2024-09-11T05:42:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 421206016 bytes 2024-09-11T05:42:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 477878272 bytes 2024-09-11T05:42:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 529569792 bytes 2024-09-11T05:42:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 586887168 bytes 2024-09-11T05:42:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 588362752 bytes 2024-09-11T05:42:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 611314688 bytes 2024-09-11T05:42:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 664690688 bytes 2024-09-11T05:42:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 695299072 bytes 2024-09-11T05:42:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 731415552 bytes 2024-09-11T05:42:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 759843840 bytes 2024-09-11T05:42:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 816519168 bytes 2024-09-11T05:42:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 842118144 bytes 2024-09-11T05:42:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 886823936 bytes 2024-09-11T05:42:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 934974464 bytes 2024-09-11T05:42:42.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 989374464 bytes 2024-09-11T05:42:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1047205888 bytes 2024-09-11T05:42:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1055652864 bytes 2024-09-11T05:42:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1103339520 bytes 2024-09-11T05:42:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1138092032 bytes 2024-09-11T05:42:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1205071872 bytes 2024-09-11T05:42:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1208096768 bytes 2024-09-11T05:42:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1316753408 bytes 2024-09-11T05:42:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1343589376 bytes 2024-09-11T05:42:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1372362752 bytes 2024-09-11T05:42:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1427299328 bytes 2024-09-11T05:42:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1483367424 bytes 2024-09-11T05:42:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1554112512 bytes 2024-09-11T05:42:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1606810624 bytes 2024-09-11T05:42:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1667481600 bytes 2024-09-11T05:42:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1680287744 bytes 2024-09-11T05:42:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1711663104 bytes 2024-09-11T05:42:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1809975296 bytes 2024-09-11T05:42:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1826058240 bytes 2024-09-11T05:42:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1840802816 bytes 2024-09-11T05:42:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1891191808 bytes 2024-09-11T05:42:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1958385664 bytes 2024-09-11T05:42:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2014609408 bytes 2024-09-11T05:42:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2061335552 bytes 2024-09-11T05:42:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2127993856 bytes 2024-09-11T05:42:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2128624640 bytes 2024-09-11T05:42:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2193132544 bytes 2024-09-11T05:42:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2193727488 bytes 2024-09-11T05:42:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2265907200 bytes 2024-09-11T05:42:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2291295232 bytes 2024-09-11T05:42:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2332254208 bytes 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:42:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-11T05:42:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-11T05:42:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-11T05:42:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_31': 2024-09-11T05:42:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13e3df3e9683 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13e3df3e9683 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:46 2024 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:46 2024 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:46 2024 2024-09-11T05:42:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 57412544-618d-4457-b99f-bd8f708b1bd6 2024-09-11T05:42:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-11T05:42:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:42:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:42:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 8943616 bytes 2024-09-11T05:42:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 11030528 bytes 2024-09-11T05:42:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 45255680 bytes 2024-09-11T05:42:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 128182272 bytes 2024-09-11T05:42:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 226253824 bytes 2024-09-11T05:42:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 287029248 bytes 2024-09-11T05:42:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 336005120 bytes 2024-09-11T05:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 395952128 bytes 2024-09-11T05:42:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 479369216 bytes 2024-09-11T05:42:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 522339328 bytes 2024-09-11T05:42:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 583384064 bytes 2024-09-11T05:42:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 663435264 bytes 2024-09-11T05:42:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 731904000 bytes 2024-09-11T05:42:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 791314432 bytes 2024-09-11T05:42:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 846394368 bytes 2024-09-11T05:42:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 891548672 bytes 2024-09-11T05:42:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 941110272 bytes 2024-09-11T05:42:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 964214784 bytes 2024-09-11T05:42:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1031769088 bytes 2024-09-11T05:42:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1105191936 bytes 2024-09-11T05:42:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1105819648 bytes 2024-09-11T05:42:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1127798784 bytes 2024-09-11T05:42:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1169745920 bytes 2024-09-11T05:42:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1198550016 bytes 2024-09-11T05:42:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1323102208 bytes 2024-09-11T05:42:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1357095936 bytes 2024-09-11T05:42:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1424805888 bytes 2024-09-11T05:42:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1462947840 bytes 2024-09-11T05:42:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1479495680 bytes 2024-09-11T05:42:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1500273664 bytes 2024-09-11T05:42:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1556288512 bytes 2024-09-11T05:42:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1593988096 bytes 2024-09-11T05:42:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1613635584 bytes 2024-09-11T05:42:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1621687296 bytes 2024-09-11T05:42:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1647333376 bytes 2024-09-11T05:42:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1693609984 bytes 2024-09-11T05:42:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1709703168 bytes 2024-09-11T05:42:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1731386368 bytes 2024-09-11T05:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1748896768 bytes 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2024-09-11T05:42:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:42:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:42:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-11T05:42:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:42:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-11T05:42:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_32': 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13eebc761ccc 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13eebc761ccc 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:53 2024 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:53 2024 2024-09-11T05:42:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:53 2024 2024-09-11T05:42:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:42:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:42:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a4730635-d6c8-461d-89b9-ef37acfa7eec 2024-09-11T05:42:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:42:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:42:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T05:42:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:42:54.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:42:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 63763456 bytes 2024-09-11T05:42:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 114537472 bytes 2024-09-11T05:42:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 212127744 bytes 2024-09-11T05:42:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 261661696 bytes 2024-09-11T05:42:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 345119744 bytes 2024-09-11T05:42:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 440943616 bytes 2024-09-11T05:42:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 459123712 bytes 2024-09-11T05:42:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 460724224 bytes 2024-09-11T05:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 567562240 bytes 2024-09-11T05:42:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 594830336 bytes 2024-09-11T05:42:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 620098560 bytes 2024-09-11T05:42:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 668531712 bytes 2024-09-11T05:42:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 681499648 bytes 2024-09-11T05:42:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 718081024 bytes 2024-09-11T05:42:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 776958976 bytes 2024-09-11T05:42:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 815695872 bytes 2024-09-11T05:42:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 829882368 bytes 2024-09-11T05:42:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 834746368 bytes 2024-09-11T05:42:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 870337536 bytes 2024-09-11T05:42:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 972900352 bytes 2024-09-11T05:42:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1024739328 bytes 2024-09-11T05:42:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1065297920 bytes 2024-09-11T05:42:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1080364032 bytes 2024-09-11T05:42:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1175925760 bytes 2024-09-11T05:42:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1196778496 bytes 2024-09-11T05:42:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1228550144 bytes 2024-09-11T05:42:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1254158336 bytes 2024-09-11T05:42:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1254789120 bytes 2024-09-11T05:42:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1357425664 bytes 2024-09-11T05:42:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1425528832 bytes 2024-09-11T05:42:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1463478272 bytes 2024-09-11T05:42:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1510359040 bytes 2024-09-11T05:43:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1537353728 bytes 2024-09-11T05:43:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1563892736 bytes 2024-09-11T05:43:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1636541440 bytes 2024-09-11T05:43:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1745721344 bytes 2024-09-11T05:43:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1775708160 bytes 2024-09-11T05:43:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1845699584 bytes 2024-09-11T05:43:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1862744064 bytes 2024-09-11T05:43:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1876093952 bytes 2024-09-11T05:43:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1898566656 bytes 2024-09-11T05:43:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1948347392 bytes 2024-09-11T05:43:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2025942016 bytes 2024-09-11T05:43:01.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2131308544 bytes 2024-09-11T05:43:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2137453568 bytes 2024-09-11T05:43:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2148585472 bytes 2024-09-11T05:43:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2188106752 bytes 2024-09-11T05:43:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2239601664 bytes 2024-09-11T05:43:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2315405312 bytes 2024-09-11T05:43:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2345490432 bytes 2024-09-11T05:43:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2388715520 bytes 2024-09-11T05:43:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2412733440 bytes 2024-09-11T05:43:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2439485440 bytes 2024-09-11T05:43:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2535952384 bytes 2024-09-11T05:43:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2567743488 bytes 2024-09-11T05:43:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2592545792 bytes 2024-09-11T05:43:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2624582656 bytes 2024-09-11T05:43:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2662016000 bytes 2024-09-11T05:43:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2691672064 bytes 2024-09-11T05:43:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2718907392 bytes 2024-09-11T05:43:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2765286400 bytes 2024-09-11T05:43:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2821397504 bytes 2024-09-11T05:43:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2857655296 bytes 2024-09-11T05:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2905303040 bytes 2024-09-11T05:43:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2928809984 bytes 2024-09-11T05:43:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2962476032 bytes 2024-09-11T05:43:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2987481088 bytes 2024-09-11T05:43:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3082414080 bytes 2024-09-11T05:43:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3107435520 bytes 2024-09-11T05:43:05.552 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.36177218625212204) > weight (0.25) 2024-09-11T05:43:05.552 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.798330203468045) > weight (0.25) 2024-09-11T05:43:05.552 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:43:05.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:43:05.553 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T05:43:05.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:43:05.554+0000 7fca1772d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 34232) UID: 1000 2024-09-11T05:43:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3220585472 bytes 2024-09-11T05:43:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3222872064 bytes 2024-09-11T05:43:05.939 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:43:05.938+0000 7f14da894640 -1 reset not still connected to 0x556435dd3ee0 2024-09-11T05:43:05.939 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:43:05.938+0000 7f14da894640 -1 reset not still connected to 0x55643b5a60d0 2024-09-11T05:43:05.939 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:43:05.938+0000 7f14da894640 -1 reset not still connected to 0x55643c1ca340 2024-09-11T05:43:05.939 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:43:05.938+0000 7f14da894640 -1 reset not still connected to 0x556441d7d1e0 2024-09-11T05:43:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3276339200 bytes 2024-09-11T05:43:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3324273664 bytes 2024-09-11T05:43:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3343464448 bytes 2024-09-11T05:43:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3411357696 bytes 2024-09-11T05:43:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3446017024 bytes 2024-09-11T05:43:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3490377728 bytes 2024-09-11T05:43:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3515042816 bytes 2024-09-11T05:43:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3521014784 bytes 2024-09-11T05:43:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3565389824 bytes 2024-09-11T05:43:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3602779136 bytes 2024-09-11T05:43:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3617569792 bytes 2024-09-11T05:43:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3677913088 bytes 2024-09-11T05:43:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3768337408 bytes 2024-09-11T05:43:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3802673152 bytes 2024-09-11T05:43:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3871905792 bytes 2024-09-11T05:43:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3905673216 bytes 2024-09-11T05:43:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3954367488 bytes 2024-09-11T05:43:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3985317888 bytes 2024-09-11T05:43:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 4050614272 bytes 2024-09-11T05:43:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 4090168320 bytes 2024-09-11T05:43:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4130082816 bytes 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-11T05:43:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:43:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-11T05:43:09.554 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:43:09.554 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:43:09.555 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:43:09.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:43:09.555 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:43:09.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:43:09.558 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-11T05:43:09.583 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25861775883036264) > weight (0.25) 2024-09-11T05:43:09.583 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5144194301901599) > weight (0.25) 2024-09-11T05:43:09.583 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6252426857837476) > weight (0.25) 2024-09-11T05:43:09.583 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5125181114943931) > weight (0.25) 2024-09-11T05:43:09.583 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-11T05:43:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:43:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_33': 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13f8fdfabb61 2024-09-11T05:43:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13f8fdfabb61 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:08 2024 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:08 2024 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:08 2024 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fe60b457-0a30-4f36-a30e-adc07ffaf32e 2024-09-11T05:43:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:43:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:43:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2024-09-11T05:43:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:43:09.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:43:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 58612736 bytes 2024-09-11T05:43:10.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.354+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1400 [3/240aff3e-f4c4-4e78-89e9-f002bb8ec31b] start: already running 2024-09-11T05:43:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 59226112 bytes 2024-09-11T05:43:10.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.374+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c000 [3/417b48dd-e251-40f1-872a-99ad8bf806db] start: already running 2024-09-11T05:43:10.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.374+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c000 [3/417b48dd-e251-40f1-872a-99ad8bf806db] start: already running 2024-09-11T05:43:10.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.446+0000 7f9e785cc640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1b80 [3/4109d16e-0d23-49cb-8378-af90cd53b465] start: already running 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c500 [3/4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c500 [3/4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e785cc640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c500 [3/4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e] start: already running 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b0a00 [3/1c1a7c61-6e2b-48cc-80d9-2dcd7ffe860a] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b0a00 [3/1c1a7c61-6e2b-48cc-80d9-2dcd7ffe860a] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14cf00 [3/7f727691-cb67-470b-9ad0-990f37a8f164] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14cf00 [3/7f727691-cb67-470b-9ad0-990f37a8f164] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c280 [3/4607f120-f3a3-4ead-9c81-2256f6efd774] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c280 [3/4607f120-f3a3-4ead-9c81-2256f6efd774] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e785cc640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c280 [3/4607f120-f3a3-4ead-9c81-2256f6efd774] start: already running 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c780 [3/db729295-01a4-4d5f-aec1-17d8b32aefee] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14c780 [3/db729295-01a4-4d5f-aec1-17d8b32aefee] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180a00 [3/d6371132-753a-4a4f-8524-8425689888b8] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180a00 [3/d6371132-753a-4a4f-8524-8425689888b8] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180000 [3/a98d5b12-6ee4-49e3-b583-ddcaf0392d73] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180000 [3/a98d5b12-6ee4-49e3-b583-ddcaf0392d73] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d680 [3/95e24eb9-a635-4d8b-b286-5bc01f045ba1] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d680 [3/95e24eb9-a635-4d8b-b286-5bc01f045ba1] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14ca00 [3/61ff442c-b39c-42e8-bcaf-1b8c56417011] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14ca00 [3/61ff442c-b39c-42e8-bcaf-1b8c56417011] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d180 [3/8de57868-2d26-4418-9611-9fe7ca6aec5d] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d180 [3/8de57868-2d26-4418-9611-9fe7ca6aec5d] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180500 [3/b16094f1-b9fc-4dcb-bceb-0331cdd63e45] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180500 [3/b16094f1-b9fc-4dcb-bceb-0331cdd63e45] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d900 [3/a4730635-d6c8-461d-89b9-ef37acfa7eec] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb14d900 [3/a4730635-d6c8-461d-89b9-ef37acfa7eec] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180780 [3/b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180780 [3/b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1680 [3/29554ac5-9f7f-47c5-8ec4-11f432ce8b61] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1680 [3/29554ac5-9f7f-47c5-8ec4-11f432ce8b61] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e785cc640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1680 [3/29554ac5-9f7f-47c5-8ec4-11f432ce8b61] start: already running 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb181180 [3/f6c4c95b-26d9-427d-9fd8-9b7fe2a1b185] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb181180 [3/f6c4c95b-26d9-427d-9fd8-9b7fe2a1b185] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180280 [3/ab674d81-64fd-4a72-a384-57c22ab78502] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5eb180280 [3/ab674d81-64fd-4a72-a384-57c22ab78502] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e84de5640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1900 [3/2c6de312-17ab-498c-82cf-a905326cbb84] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e855e6640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1900 [3/2c6de312-17ab-498c-82cf-a905326cbb84] register_admin_socket_hook: error registering admin socket commands 2024-09-11T05:43:10.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:43:10.462+0000 7f9e785cc640 -1 rbd::mirror::ImageReplayer: 0x55f5ea9b1900 [3/2c6de312-17ab-498c-82cf-a905326cbb84] start: already running 2024-09-11T05:43:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 106824704 bytes 2024-09-11T05:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 147884032 bytes 2024-09-11T05:43:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 210746368 bytes 2024-09-11T05:43:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 247272448 bytes 2024-09-11T05:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 289328128 bytes 2024-09-11T05:43:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 356647936 bytes 2024-09-11T05:43:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 402035712 bytes 2024-09-11T05:43:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 479358976 bytes 2024-09-11T05:43:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 534290432 bytes 2024-09-11T05:43:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 623194112 bytes 2024-09-11T05:43:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 658703360 bytes 2024-09-11T05:43:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 665441280 bytes 2024-09-11T05:43:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 679986176 bytes 2024-09-11T05:43:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 761623552 bytes 2024-09-11T05:43:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 777027584 bytes 2024-09-11T05:43:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 794986496 bytes 2024-09-11T05:43:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 815432704 bytes 2024-09-11T05:43:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 848509952 bytes 2024-09-11T05:43:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 870829056 bytes 2024-09-11T05:43:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 892276736 bytes 2024-09-11T05:43:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 969400320 bytes 2024-09-11T05:43:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 991162368 bytes 2024-09-11T05:43:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1024779264 bytes 2024-09-11T05:43:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1044399104 bytes 2024-09-11T05:43:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1090415616 bytes 2024-09-11T05:43:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1125358592 bytes 2024-09-11T05:43:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1214155776 bytes 2024-09-11T05:43:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1332859904 bytes 2024-09-11T05:43:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1366599680 bytes 2024-09-11T05:43:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1435117568 bytes 2024-09-11T05:43:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1446941696 bytes 2024-09-11T05:43:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1508278272 bytes 2024-09-11T05:43:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1573212160 bytes 2024-09-11T05:43:15.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1633997824 bytes 2024-09-11T05:43:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1679644672 bytes 2024-09-11T05:43:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1722035200 bytes 2024-09-11T05:43:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1779031040 bytes 2024-09-11T05:43:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1788337152 bytes 2024-09-11T05:43:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1848155136 bytes 2024-09-11T05:43:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1868788736 bytes 2024-09-11T05:43:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1956000768 bytes 2024-09-11T05:43:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2000023552 bytes 2024-09-11T05:43:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2059604992 bytes 2024-09-11T05:43:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2095175680 bytes 2024-09-11T05:43:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2211607552 bytes 2024-09-11T05:43:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2220607488 bytes 2024-09-11T05:43:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2264584192 bytes 2024-09-11T05:43:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2323010560 bytes 2024-09-11T05:43:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2423876608 bytes 2024-09-11T05:43:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2427088896 bytes 2024-09-11T05:43:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2504019968 bytes 2024-09-11T05:43:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2542278656 bytes 2024-09-11T05:43:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2573012992 bytes 2024-09-11T05:43:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2605778944 bytes 2024-09-11T05:43:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2628492288 bytes 2024-09-11T05:43:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2691415040 bytes 2024-09-11T05:43:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2743501824 bytes 2024-09-11T05:43:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2765582336 bytes 2024-09-11T05:43:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2773382144 bytes 2024-09-11T05:43:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2783438848 bytes 2024-09-11T05:43:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2843574272 bytes 2024-09-11T05:43:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2844177408 bytes 2024-09-11T05:43:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2899055616 bytes 2024-09-11T05:43:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2959169536 bytes 2024-09-11T05:43:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2980195328 bytes 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-11T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-11T05:43:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:43:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:43:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-11T05:43:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:43:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-11T05:43:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_34': 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 140828b5bf4a 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.140828b5bf4a 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:19 2024 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:19 2024 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:19 2024 2024-09-11T05:43:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:43:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:43:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 82196961-27ae-4fd0-a78e-c8801a4e74b1 2024-09-11T05:43:20.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:43:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:43:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2024-09-11T05:43:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:43:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:43:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 30806016 bytes 2024-09-11T05:43:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 34401280 bytes 2024-09-11T05:43:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 70687744 bytes 2024-09-11T05:43:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 71282688 bytes 2024-09-11T05:43:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 111271936 bytes 2024-09-11T05:43:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 116139008 bytes 2024-09-11T05:43:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 167840768 bytes 2024-09-11T05:43:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 184091648 bytes 2024-09-11T05:43:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 234279936 bytes 2024-09-11T05:43:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 341912576 bytes 2024-09-11T05:43:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 354609152 bytes 2024-09-11T05:43:22.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 362344448 bytes 2024-09-11T05:43:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 395356160 bytes 2024-09-11T05:43:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 434091008 bytes 2024-09-11T05:43:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 478697472 bytes 2024-09-11T05:43:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 495459328 bytes 2024-09-11T05:43:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 564239360 bytes 2024-09-11T05:43:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 615711744 bytes 2024-09-11T05:43:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 643853312 bytes 2024-09-11T05:43:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 680198144 bytes 2024-09-11T05:43:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 701441024 bytes 2024-09-11T05:43:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 740342784 bytes 2024-09-11T05:43:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 801203200 bytes 2024-09-11T05:43:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 828747776 bytes 2024-09-11T05:43:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 853187584 bytes 2024-09-11T05:43:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 935965696 bytes 2024-09-11T05:43:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 960654336 bytes 2024-09-11T05:43:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 976699392 bytes 2024-09-11T05:43:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 993198080 bytes 2024-09-11T05:43:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1049828352 bytes 2024-09-11T05:43:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1076903936 bytes 2024-09-11T05:43:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1099825152 bytes 2024-09-11T05:43:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1130622976 bytes 2024-09-11T05:43:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1139865600 bytes 2024-09-11T05:43:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1186557952 bytes 2024-09-11T05:43:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1194043392 bytes 2024-09-11T05:43:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1236253696 bytes 2024-09-11T05:43:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1241582592 bytes 2024-09-11T05:43:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1262617600 bytes 2024-09-11T05:43:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1285919744 bytes 2024-09-11T05:43:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1327627264 bytes 2024-09-11T05:43:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1381001216 bytes 2024-09-11T05:43:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1424734208 bytes 2024-09-11T05:43:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1449919488 bytes 2024-09-11T05:43:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1500860416 bytes 2024-09-11T05:43:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1537496064 bytes 2024-09-11T05:43:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1585343488 bytes 2024-09-11T05:43:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1585978368 bytes 2024-09-11T05:43:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1590022144 bytes 2024-09-11T05:43:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1618667520 bytes 2024-09-11T05:43:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1723885568 bytes 2024-09-11T05:43:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1777478656 bytes 2024-09-11T05:43:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1828017152 bytes 2024-09-11T05:43:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1884049408 bytes 2024-09-11T05:43:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1899014144 bytes 2024-09-11T05:43:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 1910791168 bytes 2024-09-11T05:43:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 1957797888 bytes 2024-09-11T05:43:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 1991220224 bytes 2024-09-11T05:43:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2004581376 bytes 2024-09-11T05:43:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2076899328 bytes 2024-09-11T05:43:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2111520768 bytes 2024-09-11T05:43:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2128016384 bytes 2024-09-11T05:43:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2128623616 bytes 2024-09-11T05:43:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2214153216 bytes 2024-09-11T05:43:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2246369280 bytes 2024-09-11T05:43:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2332642304 bytes 2024-09-11T05:43:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2413796352 bytes 2024-09-11T05:43:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2422043648 bytes 2024-09-11T05:43:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2470200320 bytes 2024-09-11T05:43:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2511661056 bytes 2024-09-11T05:43:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2524286976 bytes 2024-09-11T05:43:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2566845440 bytes 2024-09-11T05:43:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2592839680 bytes 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:43:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-11T05:43:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-11T05:43:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:43:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:43:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-11T05:43:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:43:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-11T05:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_35': 2024-09-11T05:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:43:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1410c3f20871 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1410c3f20871 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:31 2024 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:31 2024 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:31 2024 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4c30e781-f967-4701-9b49-4258268b118a 2024-09-11T05:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:43:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-11T05:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:43:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2024-09-11T05:43:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:43:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:43:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 118850560 bytes 2024-09-11T05:43:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 119490560 bytes 2024-09-11T05:43:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 181108736 bytes 2024-09-11T05:43:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 249331712 bytes 2024-09-11T05:43:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 291428352 bytes 2024-09-11T05:43:33.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 325131264 bytes 2024-09-11T05:43:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 326426624 bytes 2024-09-11T05:43:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 328984576 bytes 2024-09-11T05:43:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 367279104 bytes 2024-09-11T05:43:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 415289344 bytes 2024-09-11T05:43:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 523447296 bytes 2024-09-11T05:43:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 581385216 bytes 2024-09-11T05:43:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 613983232 bytes 2024-09-11T05:43:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 652278784 bytes 2024-09-11T05:43:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 673313792 bytes 2024-09-11T05:43:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 719715328 bytes 2024-09-11T05:43:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 749096960 bytes 2024-09-11T05:43:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 807298048 bytes 2024-09-11T05:43:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 867860480 bytes 2024-09-11T05:43:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 886122496 bytes 2024-09-11T05:43:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 888310784 bytes 2024-09-11T05:43:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 926597120 bytes 2024-09-11T05:43:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 936276992 bytes 2024-09-11T05:43:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 946856960 bytes 2024-09-11T05:43:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 964494336 bytes 2024-09-11T05:43:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1012993024 bytes 2024-09-11T05:43:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1064318976 bytes 2024-09-11T05:43:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1155092480 bytes 2024-09-11T05:43:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1214823424 bytes 2024-09-11T05:43:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1241962496 bytes 2024-09-11T05:43:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1334510592 bytes 2024-09-11T05:43:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1390637056 bytes 2024-09-11T05:43:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1443521536 bytes 2024-09-11T05:43:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1504287744 bytes 2024-09-11T05:43:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1505236992 bytes 2024-09-11T05:43:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1553669120 bytes 2024-09-11T05:43:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1614695424 bytes 2024-09-11T05:43:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1654289408 bytes 2024-09-11T05:43:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1727887360 bytes 2024-09-11T05:43:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1767980032 bytes 2024-09-11T05:43:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1841851392 bytes 2024-09-11T05:43:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1924310016 bytes 2024-09-11T05:43:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1963310080 bytes 2024-09-11T05:43:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1976029184 bytes 2024-09-11T05:43:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2015443968 bytes 2024-09-11T05:43:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2063811584 bytes 2024-09-11T05:43:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2128411648 bytes 2024-09-11T05:43:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2129000448 bytes 2024-09-11T05:43:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2180851712 bytes 2024-09-11T05:43:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2181463040 bytes 2024-09-11T05:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2201960448 bytes 2024-09-11T05:43:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2293652480 bytes 2024-09-11T05:43:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2302466048 bytes 2024-09-11T05:43:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2359627776 bytes 2024-09-11T05:43:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2445805568 bytes 2024-09-11T05:43:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2505910272 bytes 2024-09-11T05:43:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2537994240 bytes 2024-09-11T05:43:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2578706432 bytes 2024-09-11T05:43:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2636470272 bytes 2024-09-11T05:43:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2653227008 bytes 2024-09-11T05:43:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2730296320 bytes 2024-09-11T05:43:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2790588416 bytes 2024-09-11T05:43:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2812396544 bytes 2024-09-11T05:43:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2864659456 bytes 2024-09-11T05:43:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2886281216 bytes 2024-09-11T05:43:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2940765184 bytes 2024-09-11T05:43:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2975800320 bytes 2024-09-11T05:43:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2988789760 bytes 2024-09-11T05:43:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3022498816 bytes 2024-09-11T05:43:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3044902912 bytes 2024-09-11T05:43:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3045527552 bytes 2024-09-11T05:43:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3046146048 bytes 2024-09-11T05:43:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3076799488 bytes 2024-09-11T05:43:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3141613568 bytes 2024-09-11T05:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3189516288 bytes 2024-09-11T05:43:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3242302464 bytes 2024-09-11T05:43:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3331555328 bytes 2024-09-11T05:43:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3332168704 bytes 2024-09-11T05:43:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3366961152 bytes 2024-09-11T05:43:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3383426048 bytes 2024-09-11T05:43:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3398446080 bytes 2024-09-11T05:43:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3465768960 bytes 2024-09-11T05:43:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3481031680 bytes 2024-09-11T05:43:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3527266304 bytes 2024-09-11T05:43:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3596510208 bytes 2024-09-11T05:43:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3615584256 bytes 2024-09-11T05:43:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3616180224 bytes 2024-09-11T05:43:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3674143744 bytes 2024-09-11T05:43:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3747465216 bytes 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2024-09-11T05:43:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-11T05:43:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:43:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-11T05:43:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:43:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-11T05:43:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_36': 2024-09-11T05:43:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 141a57e469cc 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.141a57e469cc 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:45 2024 2024-09-11T05:43:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:45 2024 2024-09-11T05:43:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:45 2024 2024-09-11T05:43:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:43:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:43:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9c4bb36c-7150-40d3-a228-8a4295768e81 2024-09-11T05:43:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:43:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:43:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:43:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:43:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 27522048 bytes 2024-09-11T05:43:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 28142592 bytes 2024-09-11T05:43:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 53670912 bytes 2024-09-11T05:43:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 78503936 bytes 2024-09-11T05:43:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 116084736 bytes 2024-09-11T05:43:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 170540032 bytes 2024-09-11T05:43:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 181742592 bytes 2024-09-11T05:43:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 193590272 bytes 2024-09-11T05:43:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 221542400 bytes 2024-09-11T05:43:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 276220928 bytes 2024-09-11T05:43:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 358682624 bytes 2024-09-11T05:43:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 369959936 bytes 2024-09-11T05:43:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 439613440 bytes 2024-09-11T05:43:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 490539008 bytes 2024-09-11T05:43:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 496627712 bytes 2024-09-11T05:43:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 598796288 bytes 2024-09-11T05:43:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 648600576 bytes 2024-09-11T05:43:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 690607104 bytes 2024-09-11T05:43:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 736851968 bytes 2024-09-11T05:43:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 775785472 bytes 2024-09-11T05:43:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 837724160 bytes 2024-09-11T05:43:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 884189184 bytes 2024-09-11T05:43:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 994371584 bytes 2024-09-11T05:43:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1019897856 bytes 2024-09-11T05:43:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1069150208 bytes 2024-09-11T05:43:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1122756608 bytes 2024-09-11T05:43:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1192043520 bytes 2024-09-11T05:43:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1209619456 bytes 2024-09-11T05:43:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1233743872 bytes 2024-09-11T05:43:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1307224064 bytes 2024-09-11T05:43:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1393930240 bytes 2024-09-11T05:43:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1408290816 bytes 2024-09-11T05:43:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1517456384 bytes 2024-09-11T05:43:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1550947328 bytes 2024-09-11T05:43:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1597553664 bytes 2024-09-11T05:43:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1643516928 bytes 2024-09-11T05:43:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1671118848 bytes 2024-09-11T05:43:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1671737344 bytes 2024-09-11T05:43:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1695128576 bytes 2024-09-11T05:43:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1718002688 bytes 2024-09-11T05:43:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1774077952 bytes 2024-09-11T05:43:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1791324160 bytes 2024-09-11T05:43:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1865788416 bytes 2024-09-11T05:43:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1935060992 bytes 2024-09-11T05:43:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1946095616 bytes 2024-09-11T05:43:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2033053696 bytes 2024-09-11T05:43:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2118248448 bytes 2024-09-11T05:43:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2156070912 bytes 2024-09-11T05:43:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2253782016 bytes 2024-09-11T05:43:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2318852096 bytes 2024-09-11T05:43:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2353753088 bytes 2024-09-11T05:43:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2413837312 bytes 2024-09-11T05:43:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2429877248 bytes 2024-09-11T05:43:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2523056128 bytes 2024-09-11T05:43:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2544065536 bytes 2024-09-11T05:43:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2591131648 bytes 2024-09-11T05:43:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2620843008 bytes 2024-09-11T05:43:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2660746240 bytes 2024-09-11T05:43:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2671894528 bytes 2024-09-11T05:43:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2705973248 bytes 2024-09-11T05:43:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2739034112 bytes 2024-09-11T05:43:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:43:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:43:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:43:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:43:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:43:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-11T05:43:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:43:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:43:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-11T05:43:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:43:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_37': 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1423442a430b 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1423442a430b 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:43:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:54 2024 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:54 2024 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:54 2024 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f496dce6-4d07-4bf7-9ba7-d6c8866c01dc 2024-09-11T05:43:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-11T05:43:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:43:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:43:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:43:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:43:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:43:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:43:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:43:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 43677696 bytes 2024-09-11T05:43:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 104465408 bytes 2024-09-11T05:43:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 122891264 bytes 2024-09-11T05:43:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 151434240 bytes 2024-09-11T05:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 197149696 bytes 2024-09-11T05:43:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 243814400 bytes 2024-09-11T05:43:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 287279104 bytes 2024-09-11T05:43:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 357537792 bytes 2024-09-11T05:43:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 368870400 bytes 2024-09-11T05:43:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 424039424 bytes 2024-09-11T05:43:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 456427520 bytes 2024-09-11T05:43:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 514629632 bytes 2024-09-11T05:43:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 558543872 bytes 2024-09-11T05:43:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 601000960 bytes 2024-09-11T05:43:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 649026560 bytes 2024-09-11T05:43:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 709855232 bytes 2024-09-11T05:43:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 807227392 bytes 2024-09-11T05:43:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 906831872 bytes 2024-09-11T05:43:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1014753280 bytes 2024-09-11T05:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1025604608 bytes 2024-09-11T05:43:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1036742656 bytes 2024-09-11T05:43:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1055758336 bytes 2024-09-11T05:43:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1100647424 bytes 2024-09-11T05:43:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1118653440 bytes 2024-09-11T05:43:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1148207104 bytes 2024-09-11T05:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1160982528 bytes 2024-09-11T05:43:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1229605888 bytes 2024-09-11T05:43:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1271067648 bytes 2024-09-11T05:44:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1305762816 bytes 2024-09-11T05:44:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1306384384 bytes 2024-09-11T05:44:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1379758080 bytes 2024-09-11T05:44:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1418807296 bytes 2024-09-11T05:44:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1438065664 bytes 2024-09-11T05:44:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1475844096 bytes 2024-09-11T05:44:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1502121984 bytes 2024-09-11T05:44:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1553683456 bytes 2024-09-11T05:44:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1603129344 bytes 2024-09-11T05:44:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1647422464 bytes 2024-09-11T05:44:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1707735040 bytes 2024-09-11T05:44:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1757401088 bytes 2024-09-11T05:44:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1881763840 bytes 2024-09-11T05:44:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1882356736 bytes 2024-09-11T05:44:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1941296128 bytes 2024-09-11T05:44:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1956468736 bytes 2024-09-11T05:44:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1970658304 bytes 2024-09-11T05:44:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1987895296 bytes 2024-09-11T05:44:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2034940928 bytes 2024-09-11T05:44:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2065888256 bytes 2024-09-11T05:44:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2095707136 bytes 2024-09-11T05:44:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2119763968 bytes 2024-09-11T05:44:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2175864832 bytes 2024-09-11T05:44:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2284990464 bytes 2024-09-11T05:44:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2326144000 bytes 2024-09-11T05:44:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2368514048 bytes 2024-09-11T05:44:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2382115840 bytes 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:44:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-11T05:44:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-11T05:44:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_38': 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 142cfd504f71 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.142cfd504f71 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:03 2024 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:03 2024 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:03 2024 2024-09-11T05:44:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 62c8df86-31da-4f6f-bf55-e0f3d889691b 2024-09-11T05:44:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:44:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:44:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 67657728 bytes 2024-09-11T05:44:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 76392448 bytes 2024-09-11T05:44:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 134895616 bytes 2024-09-11T05:44:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 206239744 bytes 2024-09-11T05:44:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 215754752 bytes 2024-09-11T05:44:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 303217664 bytes 2024-09-11T05:44:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 397775872 bytes 2024-09-11T05:44:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 417626112 bytes 2024-09-11T05:44:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 502768640 bytes 2024-09-11T05:44:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 547490816 bytes 2024-09-11T05:44:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 604696576 bytes 2024-09-11T05:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 622287872 bytes 2024-09-11T05:44:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 703077376 bytes 2024-09-11T05:44:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 724482048 bytes 2024-09-11T05:44:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 764308480 bytes 2024-09-11T05:44:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 792071168 bytes 2024-09-11T05:44:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 839414784 bytes 2024-09-11T05:44:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 930493440 bytes 2024-09-11T05:44:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 954447872 bytes 2024-09-11T05:44:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 971094016 bytes 2024-09-11T05:44:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1079343104 bytes 2024-09-11T05:44:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1148905472 bytes 2024-09-11T05:44:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1232150528 bytes 2024-09-11T05:44:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1254606848 bytes 2024-09-11T05:44:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1322613760 bytes 2024-09-11T05:44:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1336962048 bytes 2024-09-11T05:44:08.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1365704704 bytes 2024-09-11T05:44:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1391650816 bytes 2024-09-11T05:44:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1420340224 bytes 2024-09-11T05:44:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1426787328 bytes 2024-09-11T05:44:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1486445568 bytes 2024-09-11T05:44:09.559 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.25355560363636087) > weight (0.25) 2024-09-11T05:44:09.559 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.690592156242824) > weight (0.25) 2024-09-11T05:44:09.559 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:44:09.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:44:09.560 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:44:09.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:44:09.561+0000 7f9535a1c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40162) UID: 1000 2024-09-11T05:44:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1501216768 bytes 2024-09-11T05:44:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1572091904 bytes 2024-09-11T05:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1601098752 bytes 2024-09-11T05:44:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1630186496 bytes 2024-09-11T05:44:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1685489664 bytes 2024-09-11T05:44:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1768394752 bytes 2024-09-11T05:44:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1770099712 bytes 2024-09-11T05:44:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1836480512 bytes 2024-09-11T05:44:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1884128256 bytes 2024-09-11T05:44:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1949905920 bytes 2024-09-11T05:44:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1994739712 bytes 2024-09-11T05:44:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2026893312 bytes 2024-09-11T05:44:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2050486272 bytes 2024-09-11T05:44:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2077809664 bytes 2024-09-11T05:44:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2122880000 bytes 2024-09-11T05:44:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-11T05:44:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-11T05:44:12.561 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:44:12.561 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:44:12.562 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:44:12.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:44:12.562 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:44:12.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:44:12.565 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-11T05:44:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-11T05:44:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_39': 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 143513fe3e8c 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.143513fe3e8c 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:11 2024 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:11 2024 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:11 2024 2024-09-11T05:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c7239aa6-f45d-4b14-a8bb-4c545912a4a8 2024-09-11T05:44:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:44:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 610304 bytes 2024-09-11T05:44:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 20135936 bytes 2024-09-11T05:44:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 50161664 bytes 2024-09-11T05:44:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 96776192 bytes 2024-09-11T05:44:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 154770432 bytes 2024-09-11T05:44:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 187317248 bytes 2024-09-11T05:44:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 223638528 bytes 2024-09-11T05:44:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 287352832 bytes 2024-09-11T05:44:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 351700992 bytes 2024-09-11T05:44:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 376534016 bytes 2024-09-11T05:44:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 397156352 bytes 2024-09-11T05:44:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 437523456 bytes 2024-09-11T05:44:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 487920640 bytes 2024-09-11T05:44:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 526000128 bytes 2024-09-11T05:44:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 564691968 bytes 2024-09-11T05:44:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 655924224 bytes 2024-09-11T05:44:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 670899200 bytes 2024-09-11T05:44:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 671503360 bytes 2024-09-11T05:44:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 785396736 bytes 2024-09-11T05:44:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 861707264 bytes 2024-09-11T05:44:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 885942272 bytes 2024-09-11T05:44:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 940457984 bytes 2024-09-11T05:44:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1004983296 bytes 2024-09-11T05:44:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1071761408 bytes 2024-09-11T05:44:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1113729024 bytes 2024-09-11T05:44:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1141131264 bytes 2024-09-11T05:44:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1159058432 bytes 2024-09-11T05:44:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1212211200 bytes 2024-09-11T05:44:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1255704576 bytes 2024-09-11T05:44:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1293154304 bytes 2024-09-11T05:44:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1345283072 bytes 2024-09-11T05:44:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1425673216 bytes 2024-09-11T05:44:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1468554240 bytes 2024-09-11T05:44:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1506852864 bytes 2024-09-11T05:44:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1544380416 bytes 2024-09-11T05:44:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1545041920 bytes 2024-09-11T05:44:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1609638912 bytes 2024-09-11T05:44:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1689183232 bytes 2024-09-11T05:44:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1712010240 bytes 2024-09-11T05:44:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1742061568 bytes 2024-09-11T05:44:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1769868288 bytes 2024-09-11T05:44:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1803907072 bytes 2024-09-11T05:44:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1823619072 bytes 2024-09-11T05:44:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1853778944 bytes 2024-09-11T05:44:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1951835136 bytes 2024-09-11T05:44:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2008702976 bytes 2024-09-11T05:44:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2057539584 bytes 2024-09-11T05:44:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2111147008 bytes 2024-09-11T05:44:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2148221952 bytes 2024-09-11T05:44:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2162070528 bytes 2024-09-11T05:44:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2220517376 bytes 2024-09-11T05:44:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2255164416 bytes 2024-09-11T05:44:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2339877888 bytes 2024-09-11T05:44:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-11T05:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-11T05:44:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_40': 2024-09-11T05:44:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 14463e7ea465 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.14463e7ea465 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:20 2024 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:20 2024 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:20 2024 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d6b90507-14e6-446f-bf0f-ecb9a6a2cc29 2024-09-11T05:44:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-11T05:44:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:44:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:44:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 37445632 bytes 2024-09-11T05:44:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 95214592 bytes 2024-09-11T05:44:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 127904768 bytes 2024-09-11T05:44:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 200951808 bytes 2024-09-11T05:44:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 273255424 bytes 2024-09-11T05:44:22.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 345553920 bytes 2024-09-11T05:44:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 397585408 bytes 2024-09-11T05:44:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 453035008 bytes 2024-09-11T05:44:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 520377344 bytes 2024-09-11T05:44:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 559453184 bytes 2024-09-11T05:44:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 595619840 bytes 2024-09-11T05:44:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 634949632 bytes 2024-09-11T05:44:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 677306368 bytes 2024-09-11T05:44:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 767518720 bytes 2024-09-11T05:44:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 795707392 bytes 2024-09-11T05:44:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 838513664 bytes 2024-09-11T05:44:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 862002176 bytes 2024-09-11T05:44:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 871986176 bytes 2024-09-11T05:44:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 884794368 bytes 2024-09-11T05:44:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 918446080 bytes 2024-09-11T05:44:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 956208128 bytes 2024-09-11T05:44:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 973591552 bytes 2024-09-11T05:44:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1048282112 bytes 2024-09-11T05:44:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1082643456 bytes 2024-09-11T05:44:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1083274240 bytes 2024-09-11T05:44:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1087143936 bytes 2024-09-11T05:44:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1115790336 bytes 2024-09-11T05:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1169084416 bytes 2024-09-11T05:44:26.007 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:44:26.009+0000 7f14da894640 -1 reset not still connected to 0x556435dd3ee0 2024-09-11T05:44:26.007 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:44:26.009+0000 7f14da894640 -1 reset not still connected to 0x55643b5a60d0 2024-09-11T05:44:26.007 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:44:26.009+0000 7f14da894640 -1 reset not still connected to 0x55643c1ca340 2024-09-11T05:44:26.008 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:44:26.009+0000 7f14da894640 -1 reset not still connected to 0x556441d7d1e0 2024-09-11T05:44:26.012 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2024-09-11T05:44:26.013+0000 7f14da894640 -1 reset not still connected to 0x55644dea85b0 2024-09-11T05:44:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1199574016 bytes 2024-09-11T05:44:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1214327808 bytes 2024-09-11T05:44:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1295926272 bytes 2024-09-11T05:44:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1296534528 bytes 2024-09-11T05:44:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1307919360 bytes 2024-09-11T05:44:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1382860800 bytes 2024-09-11T05:44:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1452727296 bytes 2024-09-11T05:44:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1486523392 bytes 2024-09-11T05:44:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1524692992 bytes 2024-09-11T05:44:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1532926976 bytes 2024-09-11T05:44:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1579565056 bytes 2024-09-11T05:44:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1659555840 bytes 2024-09-11T05:44:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1695300608 bytes 2024-09-11T05:44:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1751291904 bytes 2024-09-11T05:44:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1794992128 bytes 2024-09-11T05:44:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1817249792 bytes 2024-09-11T05:44:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1827295232 bytes 2024-09-11T05:44:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1835425792 bytes 2024-09-11T05:44:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1883233280 bytes 2024-09-11T05:44:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1955526656 bytes 2024-09-11T05:44:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2063174656 bytes 2024-09-11T05:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2099008512 bytes 2024-09-11T05:44:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2126556160 bytes 2024-09-11T05:44:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2167490560 bytes 2024-09-11T05:44:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2200145920 bytes 2024-09-11T05:44:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2200745984 bytes 2024-09-11T05:44:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2247542784 bytes 2024-09-11T05:44:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2253786112 bytes 2024-09-11T05:44:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2358175744 bytes 2024-09-11T05:44:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2392543232 bytes 2024-09-11T05:44:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2436681728 bytes 2024-09-11T05:44:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2461334528 bytes 2024-09-11T05:44:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2533592064 bytes 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-11T05:44:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-11T05:44:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_41': 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 14506e02b2ac 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.14506e02b2ac 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:30 2024 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:30 2024 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:30 2024 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f6dadc26-b728-4e0a-b9ab-f8acb9551249 2024-09-11T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-11T05:44:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:44:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:44:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 33667072 bytes 2024-09-11T05:44:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 100200448 bytes 2024-09-11T05:44:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 157407232 bytes 2024-09-11T05:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 193606656 bytes 2024-09-11T05:44:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 252077056 bytes 2024-09-11T05:44:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 325080064 bytes 2024-09-11T05:44:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 437785600 bytes 2024-09-11T05:44:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 478005248 bytes 2024-09-11T05:44:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 503318528 bytes 2024-09-11T05:44:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 561186816 bytes 2024-09-11T05:44:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 615767040 bytes 2024-09-11T05:44:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 627162112 bytes 2024-09-11T05:44:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 686786560 bytes 2024-09-11T05:44:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 750457856 bytes 2024-09-11T05:44:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 777287680 bytes 2024-09-11T05:44:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 805924864 bytes 2024-09-11T05:44:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 841543680 bytes 2024-09-11T05:44:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 865542144 bytes 2024-09-11T05:44:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 928601088 bytes 2024-09-11T05:44:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 989511680 bytes 2024-09-11T05:44:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1031641088 bytes 2024-09-11T05:44:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1047327744 bytes 2024-09-11T05:44:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1078167552 bytes 2024-09-11T05:44:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1129612288 bytes 2024-09-11T05:44:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1218574336 bytes 2024-09-11T05:44:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1284986880 bytes 2024-09-11T05:44:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1347868672 bytes 2024-09-11T05:44:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1423370240 bytes 2024-09-11T05:44:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1488053248 bytes 2024-09-11T05:44:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1527310336 bytes 2024-09-11T05:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1550325760 bytes 2024-09-11T05:44:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1610303488 bytes 2024-09-11T05:44:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1640705024 bytes 2024-09-11T05:44:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1687123968 bytes 2024-09-11T05:44:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1806755840 bytes 2024-09-11T05:44:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1847087104 bytes 2024-09-11T05:44:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1877413888 bytes 2024-09-11T05:44:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1934579712 bytes 2024-09-11T05:44:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1968496640 bytes 2024-09-11T05:44:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1997387776 bytes 2024-09-11T05:44:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2051195904 bytes 2024-09-11T05:44:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2100079616 bytes 2024-09-11T05:44:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2161867776 bytes 2024-09-11T05:44:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2208299008 bytes 2024-09-11T05:44:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2235786240 bytes 2024-09-11T05:44:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2285174784 bytes 2024-09-11T05:44:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2310144000 bytes 2024-09-11T05:44:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2423692288 bytes 2024-09-11T05:44:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2550742016 bytes 2024-09-11T05:44:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2589502464 bytes 2024-09-11T05:44:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2625583104 bytes 2024-09-11T05:44:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2662785024 bytes 2024-09-11T05:44:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2667481088 bytes 2024-09-11T05:44:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2723043328 bytes 2024-09-11T05:44:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2739538944 bytes 2024-09-11T05:44:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2792429568 bytes 2024-09-11T05:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2833311744 bytes 2024-09-11T05:44:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2837210112 bytes 2024-09-11T05:44:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2852415488 bytes 2024-09-11T05:44:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2888342528 bytes 2024-09-11T05:44:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2888962048 bytes 2024-09-11T05:44:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2915337216 bytes 2024-09-11T05:44:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2962226176 bytes 2024-09-11T05:44:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 3012722688 bytes 2024-09-11T05:44:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3032876032 bytes 2024-09-11T05:44:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3061387264 bytes 2024-09-11T05:44:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3120218112 bytes 2024-09-11T05:44:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3178767360 bytes 2024-09-11T05:44:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3203338240 bytes 2024-09-11T05:44:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3252299776 bytes 2024-09-11T05:44:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3299638272 bytes 2024-09-11T05:44:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3316851712 bytes 2024-09-11T05:44:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3327738880 bytes 2024-09-11T05:44:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3400815616 bytes 2024-09-11T05:44:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3416551424 bytes 2024-09-11T05:44:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3479628800 bytes 2024-09-11T05:44:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3543268352 bytes 2024-09-11T05:44:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3602566144 bytes 2024-09-11T05:44:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3631105024 bytes 2024-09-11T05:44:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3649645568 bytes 2024-09-11T05:44:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-11T05:44:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-11T05:44:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-11T05:44:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_42': 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 145842266385 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.145842266385 2024-09-11T05:44:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:43 2024 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:43 2024 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:43 2024 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 31052fdc-4c34-4a83-9382-8514a3004673 2024-09-11T05:44:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:44:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:44:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 66909184 bytes 2024-09-11T05:44:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 84760576 bytes 2024-09-11T05:44:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 114545664 bytes 2024-09-11T05:44:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 136071168 bytes 2024-09-11T05:44:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 228956160 bytes 2024-09-11T05:44:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 275863552 bytes 2024-09-11T05:44:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 315656192 bytes 2024-09-11T05:44:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 355740672 bytes 2024-09-11T05:44:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 380578816 bytes 2024-09-11T05:44:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 444265472 bytes 2024-09-11T05:44:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 466180096 bytes 2024-09-11T05:44:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 484231168 bytes 2024-09-11T05:44:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 547563520 bytes 2024-09-11T05:44:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 599925760 bytes 2024-09-11T05:44:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 620165120 bytes 2024-09-11T05:44:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 656304128 bytes 2024-09-11T05:44:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 668800000 bytes 2024-09-11T05:44:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 757200896 bytes 2024-09-11T05:44:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 817855488 bytes 2024-09-11T05:44:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 885853184 bytes 2024-09-11T05:44:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 956934144 bytes 2024-09-11T05:44:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 990185472 bytes 2024-09-11T05:44:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1021985792 bytes 2024-09-11T05:44:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1069283328 bytes 2024-09-11T05:44:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1140263936 bytes 2024-09-11T05:44:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1153225728 bytes 2024-09-11T05:44:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1161385984 bytes 2024-09-11T05:44:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1197776896 bytes 2024-09-11T05:44:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1266471936 bytes 2024-09-11T05:44:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1331162112 bytes 2024-09-11T05:44:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1378114560 bytes 2024-09-11T05:44:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1396661248 bytes 2024-09-11T05:44:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1460641792 bytes 2024-09-11T05:44:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1461274624 bytes 2024-09-11T05:44:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1514058752 bytes 2024-09-11T05:44:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1552745472 bytes 2024-09-11T05:44:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1632239616 bytes 2024-09-11T05:44:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1658617856 bytes 2024-09-11T05:44:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1709477888 bytes 2024-09-11T05:44:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1757665280 bytes 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2024-09-11T05:44:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:44:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:44:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-11T05:44:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:44:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_43': 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1461df7d4d80 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1461df7d4d80 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:44:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:50 2024 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:50 2024 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:50 2024 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d388f747-dd90-4450-9037-62799125d163 2024-09-11T05:44:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:44:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:44:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2024-09-11T05:44:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:44:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:44:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 71099392 bytes 2024-09-11T05:44:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 116364288 bytes 2024-09-11T05:44:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 207915008 bytes 2024-09-11T05:44:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 249928704 bytes 2024-09-11T05:44:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 275632128 bytes 2024-09-11T05:44:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 300686336 bytes 2024-09-11T05:44:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 315073536 bytes 2024-09-11T05:44:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 342573056 bytes 2024-09-11T05:44:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 412164096 bytes 2024-09-11T05:44:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 505622528 bytes 2024-09-11T05:44:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 506213376 bytes 2024-09-11T05:44:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 523507712 bytes 2024-09-11T05:44:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 592992256 bytes 2024-09-11T05:44:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 612292608 bytes 2024-09-11T05:44:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 642004992 bytes 2024-09-11T05:44:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 722229248 bytes 2024-09-11T05:44:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 765651968 bytes 2024-09-11T05:44:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 812858368 bytes 2024-09-11T05:44:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 813485056 bytes 2024-09-11T05:44:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 839769088 bytes 2024-09-11T05:44:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 904988672 bytes 2024-09-11T05:44:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 910106624 bytes 2024-09-11T05:44:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 997757952 bytes 2024-09-11T05:44:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1051944960 bytes 2024-09-11T05:44:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1093830656 bytes 2024-09-11T05:44:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1145204736 bytes 2024-09-11T05:44:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1177416704 bytes 2024-09-11T05:44:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1202851840 bytes 2024-09-11T05:44:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1213993984 bytes 2024-09-11T05:44:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1287103488 bytes 2024-09-11T05:44:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1342623744 bytes 2024-09-11T05:44:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1411838976 bytes 2024-09-11T05:44:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1480288256 bytes 2024-09-11T05:44:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1526549504 bytes 2024-09-11T05:44:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1579579392 bytes 2024-09-11T05:44:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1657251840 bytes 2024-09-11T05:44:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1732785152 bytes 2024-09-11T05:44:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1733373952 bytes 2024-09-11T05:44:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1831536640 bytes 2024-09-11T05:44:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1882041344 bytes 2024-09-11T05:44:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1921265664 bytes 2024-09-11T05:44:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1921880064 bytes 2024-09-11T05:44:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1977885696 bytes 2024-09-11T05:44:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1991495680 bytes 2024-09-11T05:44:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2012570624 bytes 2024-09-11T05:44:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2074489856 bytes 2024-09-11T05:44:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2097881088 bytes 2024-09-11T05:44:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2158082048 bytes 2024-09-11T05:44:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2190372864 bytes 2024-09-11T05:44:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2245490688 bytes 2024-09-11T05:44:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2287355904 bytes 2024-09-11T05:44:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2314384384 bytes 2024-09-11T05:44:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2314993664 bytes 2024-09-11T05:44:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2382386176 bytes 2024-09-11T05:44:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2423419904 bytes 2024-09-11T05:45:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2446846976 bytes 2024-09-11T05:45:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2449763328 bytes 2024-09-11T05:45:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2531321856 bytes 2024-09-11T05:45:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2609756160 bytes 2024-09-11T05:45:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2642202624 bytes 2024-09-11T05:45:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2669358080 bytes 2024-09-11T05:45:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-11T05:45:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-11T05:45:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-11T05:45:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_44': 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 146a3cc57460 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.146a3cc57460 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:00 2024 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:00 2024 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:00 2024 2024-09-11T05:45:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0af1ac05-71b0-44c5-9afe-6c451a66416b 2024-09-11T05:45:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:45:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T05:45:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 49508352 bytes 2024-09-11T05:45:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 81636352 bytes 2024-09-11T05:45:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 95804416 bytes 2024-09-11T05:45:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 174579712 bytes 2024-09-11T05:45:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 242101248 bytes 2024-09-11T05:45:02.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 291298304 bytes 2024-09-11T05:45:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 360089600 bytes 2024-09-11T05:45:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 407558144 bytes 2024-09-11T05:45:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 453397504 bytes 2024-09-11T05:45:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 525199360 bytes 2024-09-11T05:45:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 527826944 bytes 2024-09-11T05:45:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 539759616 bytes 2024-09-11T05:45:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 549840896 bytes 2024-09-11T05:45:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 572973056 bytes 2024-09-11T05:45:04.082 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:45:04.080+0000 7f3afe201640 -1 reset not still connected to 0x5651eb8a96c0 2024-09-11T05:45:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 654407680 bytes 2024-09-11T05:45:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 705890304 bytes 2024-09-11T05:45:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 717104128 bytes 2024-09-11T05:45:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 751142912 bytes 2024-09-11T05:45:04.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 805769216 bytes 2024-09-11T05:45:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 856569856 bytes 2024-09-11T05:45:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 882565120 bytes 2024-09-11T05:45:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 907745280 bytes 2024-09-11T05:45:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 954923008 bytes 2024-09-11T05:45:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1028977664 bytes 2024-09-11T05:45:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1104351232 bytes 2024-09-11T05:45:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1133547520 bytes 2024-09-11T05:45:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1219206144 bytes 2024-09-11T05:45:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1258638336 bytes 2024-09-11T05:45:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1332509696 bytes 2024-09-11T05:45:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1358608384 bytes 2024-09-11T05:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1398148096 bytes 2024-09-11T05:45:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1452177408 bytes 2024-09-11T05:45:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1506329600 bytes 2024-09-11T05:45:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1540365312 bytes 2024-09-11T05:45:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1585143808 bytes 2024-09-11T05:45:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1632496640 bytes 2024-09-11T05:45:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1664622592 bytes 2024-09-11T05:45:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1681424384 bytes 2024-09-11T05:45:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1711806464 bytes 2024-09-11T05:45:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1779955712 bytes 2024-09-11T05:45:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1794275328 bytes 2024-09-11T05:45:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1828745216 bytes 2024-09-11T05:45:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1888155648 bytes 2024-09-11T05:45:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1935712256 bytes 2024-09-11T05:45:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1964551168 bytes 2024-09-11T05:45:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1965178880 bytes 2024-09-11T05:45:08.584 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:45:08.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:45:08.585 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T05:45:08.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:45:08.588+0000 7f539741b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38569) UID: 1000 2024-09-11T05:45:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2006782976 bytes 2024-09-11T05:45:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2030495744 bytes 2024-09-11T05:45:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2117628928 bytes 2024-09-11T05:45:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2169259008 bytes 2024-09-11T05:45:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2206854144 bytes 2024-09-11T05:45:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2254415872 bytes 2024-09-11T05:45:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2255379456 bytes 2024-09-11T05:45:09.586 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:45:09.586 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:45:09.586 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:45:09.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:45:09.587 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:45:09.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:45:09.589 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-11T05:45:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2299770880 bytes 2024-09-11T05:45:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:09.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-11T05:45:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_45': 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 147ada3db9ba 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.147ada3db9ba 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:09 2024 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:09 2024 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:09 2024 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: eec72d56-6d39-496e-a649-5ff1b2b56355 2024-09-11T05:45:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:45:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T05:45:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 17185792 bytes 2024-09-11T05:45:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 52408320 bytes 2024-09-11T05:45:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 83750912 bytes 2024-09-11T05:45:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 131064832 bytes 2024-09-11T05:45:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 225872896 bytes 2024-09-11T05:45:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 271286272 bytes 2024-09-11T05:45:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 330560512 bytes 2024-09-11T05:45:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 361729024 bytes 2024-09-11T05:45:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 393590784 bytes 2024-09-11T05:45:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 441740288 bytes 2024-09-11T05:45:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 513975296 bytes 2024-09-11T05:45:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 535469056 bytes 2024-09-11T05:45:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 555540480 bytes 2024-09-11T05:45:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 582119424 bytes 2024-09-11T05:45:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 595329024 bytes 2024-09-11T05:45:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 632638464 bytes 2024-09-11T05:45:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 698202112 bytes 2024-09-11T05:45:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 738155520 bytes 2024-09-11T05:45:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 738775040 bytes 2024-09-11T05:45:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 814674944 bytes 2024-09-11T05:45:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 815295488 bytes 2024-09-11T05:45:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 848613376 bytes 2024-09-11T05:45:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 893372416 bytes 2024-09-11T05:45:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 950917120 bytes 2024-09-11T05:45:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 986579968 bytes 2024-09-11T05:45:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1058743296 bytes 2024-09-11T05:45:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1100624896 bytes 2024-09-11T05:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1123023872 bytes 2024-09-11T05:45:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1144715264 bytes 2024-09-11T05:45:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1223437312 bytes 2024-09-11T05:45:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1282619392 bytes 2024-09-11T05:45:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1367361536 bytes 2024-09-11T05:45:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-11T05:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-11T05:45:16.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-11T05:45:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_46': 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 14842a0efac3 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.14842a0efac3 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:15 2024 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:15 2024 2024-09-11T05:45:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:15 2024 2024-09-11T05:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 854d50c4-dd91-4b99-9272-800916647ca1 2024-09-11T05:45:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-11T05:45:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:45:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2024-09-11T05:45:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 64627712 bytes 2024-09-11T05:45:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 126107648 bytes 2024-09-11T05:45:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 149299200 bytes 2024-09-11T05:45:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 186249216 bytes 2024-09-11T05:45:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 254516224 bytes 2024-09-11T05:45:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 257640448 bytes 2024-09-11T05:45:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 295860224 bytes 2024-09-11T05:45:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 336000000 bytes 2024-09-11T05:45:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 354486272 bytes 2024-09-11T05:45:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 403944448 bytes 2024-09-11T05:45:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 415278080 bytes 2024-09-11T05:45:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 431408128 bytes 2024-09-11T05:45:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 475671552 bytes 2024-09-11T05:45:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 505713664 bytes 2024-09-11T05:45:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 532842496 bytes 2024-09-11T05:45:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 543401984 bytes 2024-09-11T05:45:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 577098752 bytes 2024-09-11T05:45:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 617955328 bytes 2024-09-11T05:45:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 670668800 bytes 2024-09-11T05:45:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 722853888 bytes 2024-09-11T05:45:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 768089088 bytes 2024-09-11T05:45:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 807711744 bytes 2024-09-11T05:45:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 842817536 bytes 2024-09-11T05:45:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 859457536 bytes 2024-09-11T05:45:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 921251840 bytes 2024-09-11T05:45:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 958117888 bytes 2024-09-11T05:45:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 984902656 bytes 2024-09-11T05:45:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1037926400 bytes 2024-09-11T05:45:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1073301504 bytes 2024-09-11T05:45:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1115055104 bytes 2024-09-11T05:45:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1119452160 bytes 2024-09-11T05:45:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1120055296 bytes 2024-09-11T05:45:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1147375616 bytes 2024-09-11T05:45:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1159379968 bytes 2024-09-11T05:45:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1213215744 bytes 2024-09-11T05:45:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1213834240 bytes 2024-09-11T05:45:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1239882752 bytes 2024-09-11T05:45:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1257623552 bytes 2024-09-11T05:45:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1395348480 bytes 2024-09-11T05:45:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1427507200 bytes 2024-09-11T05:45:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1501762560 bytes 2024-09-11T05:45:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1519240192 bytes 2024-09-11T05:45:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1592647680 bytes 2024-09-11T05:45:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1624119296 bytes 2024-09-11T05:45:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1661718528 bytes 2024-09-11T05:45:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1695835136 bytes 2024-09-11T05:45:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1732814848 bytes 2024-09-11T05:45:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1745815552 bytes 2024-09-11T05:45:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1757748224 bytes 2024-09-11T05:45:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1836805120 bytes 2024-09-11T05:45:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:45:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:45:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-11T05:45:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-11T05:45:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_47': 2024-09-11T05:45:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 148d9c2c56ef 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.148d9c2c56ef 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:23 2024 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:23 2024 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:23 2024 2024-09-11T05:45:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9e3a1edf-65b1-4acf-aa6f-d0e474a04d1d 2024-09-11T05:45:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:45:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T05:45:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 18603008 bytes 2024-09-11T05:45:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 78624768 bytes 2024-09-11T05:45:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 93723648 bytes 2024-09-11T05:45:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 116814848 bytes 2024-09-11T05:45:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 123762688 bytes 2024-09-11T05:45:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 156652544 bytes 2024-09-11T05:45:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 188938240 bytes 2024-09-11T05:45:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 214601728 bytes 2024-09-11T05:45:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 254359552 bytes 2024-09-11T05:45:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 357933056 bytes 2024-09-11T05:45:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 374612992 bytes 2024-09-11T05:45:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 406812672 bytes 2024-09-11T05:45:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 436669440 bytes 2024-09-11T05:45:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 507621376 bytes 2024-09-11T05:45:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 537877504 bytes 2024-09-11T05:45:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 603197440 bytes 2024-09-11T05:45:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 644197376 bytes 2024-09-11T05:45:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 746932224 bytes 2024-09-11T05:45:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 767506432 bytes 2024-09-11T05:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 791515136 bytes 2024-09-11T05:45:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 853918720 bytes 2024-09-11T05:45:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 876504064 bytes 2024-09-11T05:45:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 943037440 bytes 2024-09-11T05:45:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 948221952 bytes 2024-09-11T05:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 973877248 bytes 2024-09-11T05:45:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1055433728 bytes 2024-09-11T05:45:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1098065920 bytes 2024-09-11T05:45:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1142504448 bytes 2024-09-11T05:45:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1220826112 bytes 2024-09-11T05:45:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1283506176 bytes 2024-09-11T05:45:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1323743232 bytes 2024-09-11T05:45:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1344787456 bytes 2024-09-11T05:45:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1378651136 bytes 2024-09-11T05:45:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1403058176 bytes 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:45:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-11T05:45:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_48': 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1495c56008b 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1495c56008b 2024-09-11T05:45:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:29 2024 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:29 2024 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:29 2024 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5a84d065-9a68-4a66-a745-37e33ab34e73 2024-09-11T05:45:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:45:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T05:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 109753344 bytes 2024-09-11T05:45:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 142185472 bytes 2024-09-11T05:45:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 206555136 bytes 2024-09-11T05:45:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 251838464 bytes 2024-09-11T05:45:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 300291072 bytes 2024-09-11T05:45:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 328865792 bytes 2024-09-11T05:45:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 364624896 bytes 2024-09-11T05:45:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 376169472 bytes 2024-09-11T05:45:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 404302848 bytes 2024-09-11T05:45:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 415244288 bytes 2024-09-11T05:45:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 485099520 bytes 2024-09-11T05:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 567932928 bytes 2024-09-11T05:45:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 620675072 bytes 2024-09-11T05:45:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 672604160 bytes 2024-09-11T05:45:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 690245632 bytes 2024-09-11T05:45:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 726077440 bytes 2024-09-11T05:45:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 772344832 bytes 2024-09-11T05:45:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 920274944 bytes 2024-09-11T05:45:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 942839808 bytes 2024-09-11T05:45:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1020930048 bytes 2024-09-11T05:45:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1043058688 bytes 2024-09-11T05:45:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1115247616 bytes 2024-09-11T05:45:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1137972224 bytes 2024-09-11T05:45:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1176866816 bytes 2024-09-11T05:45:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1222571008 bytes 2024-09-11T05:45:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1266411520 bytes 2024-09-11T05:45:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1292905472 bytes 2024-09-11T05:45:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1354811392 bytes 2024-09-11T05:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1379221504 bytes 2024-09-11T05:45:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1399142400 bytes 2024-09-11T05:45:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1461157888 bytes 2024-09-11T05:45:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1489705984 bytes 2024-09-11T05:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-11T05:45:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-11T05:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-11T05:45:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_49': 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 149ea43983cd 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.149ea43983cd 2024-09-11T05:45:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:36 2024 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:36 2024 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:36 2024 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c3b8fd00-067d-4de2-881a-8a00643e7a65 2024-09-11T05:45:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:45:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:45:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 47858688 bytes 2024-09-11T05:45:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 74634240 bytes 2024-09-11T05:45:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 119105536 bytes 2024-09-11T05:45:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 184994816 bytes 2024-09-11T05:45:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 218586112 bytes 2024-09-11T05:45:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 233007104 bytes 2024-09-11T05:45:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 355949568 bytes 2024-09-11T05:45:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 409294848 bytes 2024-09-11T05:45:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 461512704 bytes 2024-09-11T05:45:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 462138368 bytes 2024-09-11T05:45:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 477864960 bytes 2024-09-11T05:45:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 529143808 bytes 2024-09-11T05:45:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 544039936 bytes 2024-09-11T05:45:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 574087168 bytes 2024-09-11T05:45:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 582413312 bytes 2024-09-11T05:45:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 605694976 bytes 2024-09-11T05:45:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 655877120 bytes 2024-09-11T05:45:39.426 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:45:39.424+0000 7f3afe201640 -1 reset not still connected to 0x5651f3b89c70 2024-09-11T05:45:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 708816896 bytes 2024-09-11T05:45:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 739200000 bytes 2024-09-11T05:45:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 783090688 bytes 2024-09-11T05:45:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 803655680 bytes 2024-09-11T05:45:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 818160640 bytes 2024-09-11T05:45:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 827767808 bytes 2024-09-11T05:45:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 891820032 bytes 2024-09-11T05:45:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 928155648 bytes 2024-09-11T05:45:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 955283456 bytes 2024-09-11T05:45:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 988193792 bytes 2024-09-11T05:45:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 989660160 bytes 2024-09-11T05:45:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1058819072 bytes 2024-09-11T05:45:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1076639744 bytes 2024-09-11T05:45:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1107189760 bytes 2024-09-11T05:45:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1141536768 bytes 2024-09-11T05:45:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1173778432 bytes 2024-09-11T05:45:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1267937280 bytes 2024-09-11T05:45:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1341181952 bytes 2024-09-11T05:45:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1364572160 bytes 2024-09-11T05:45:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1417955328 bytes 2024-09-11T05:45:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1439373312 bytes 2024-09-11T05:45:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1451760640 bytes 2024-09-11T05:45:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1472764928 bytes 2024-09-11T05:45:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1534781440 bytes 2024-09-11T05:45:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1577852928 bytes 2024-09-11T05:45:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n snapshot ']' 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 128M ']' 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:45:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' image = image ']' 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-11T05:45:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:45:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:45:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-11T05:45:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_50': 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 14a84d71f01 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.14a84d71f01 2024-09-11T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:43 2024 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:43 2024 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:43 2024 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2f74885b-b07d-4f8f-aa98-e0b0749a2019 2024-09-11T05:45:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:45:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n '' ']' 2024-09-11T05:45:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-11T05:45:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:45:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:45:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:45:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:45:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:45:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 24884224 bytes 2024-09-11T05:45:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 62924800 bytes 2024-09-11T05:45:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 171187200 bytes 2024-09-11T05:45:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 186930176 bytes 2024-09-11T05:45:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 215232512 bytes 2024-09-11T05:45:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 236795904 bytes 2024-09-11T05:45:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 287252480 bytes 2024-09-11T05:45:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 330727424 bytes 2024-09-11T05:45:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 350319616 bytes 2024-09-11T05:45:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 409656320 bytes 2024-09-11T05:45:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 470178816 bytes 2024-09-11T05:45:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 470794240 bytes 2024-09-11T05:45:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 509412352 bytes 2024-09-11T05:45:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 544715776 bytes 2024-09-11T05:45:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 585615360 bytes 2024-09-11T05:45:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 629409792 bytes 2024-09-11T05:45:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 652884992 bytes 2024-09-11T05:45:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 716794880 bytes 2024-09-11T05:45:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 777970688 bytes 2024-09-11T05:45:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 839577600 bytes 2024-09-11T05:45:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 878787584 bytes 2024-09-11T05:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 883444736 bytes 2024-09-11T05:45:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 909385728 bytes 2024-09-11T05:45:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 952761344 bytes 2024-09-11T05:45:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 977517568 bytes 2024-09-11T05:45:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 987552768 bytes 2024-09-11T05:45:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1056599040 bytes 2024-09-11T05:45:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1098158080 bytes 2024-09-11T05:45:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1119433728 bytes 2024-09-11T05:45:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1213984768 bytes 2024-09-11T05:45:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1245504512 bytes 2024-09-11T05:45:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1297490944 bytes 2024-09-11T05:45:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1314080768 bytes 2024-09-11T05:45:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1327585280 bytes 2024-09-11T05:45:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1363977216 bytes 2024-09-11T05:45:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1394863104 bytes 2024-09-11T05:45:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1452875776 bytes 2024-09-11T05:45:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1506103296 bytes 2024-09-11T05:45:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1535745024 bytes 2024-09-11T05:45:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1582163968 bytes 2024-09-11T05:45:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1587150848 bytes 2024-09-11T05:45:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1653824512 bytes 2024-09-11T05:45:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1701968896 bytes 2024-09-11T05:45:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1772527616 bytes 2024-09-11T05:45:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1872463872 bytes 2024-09-11T05:45:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1873084416 bytes 2024-09-11T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1902482432 bytes 2024-09-11T05:45:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1933015040 bytes 2024-09-11T05:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1959389184 bytes 2024-09-11T05:45:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2024462336 bytes 2024-09-11T05:45:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2062767104 bytes 2024-09-11T05:45:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2107466752 bytes 2024-09-11T05:45:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2187182080 bytes 2024-09-11T05:45:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2275544064 bytes 2024-09-11T05:45:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2341097472 bytes 2024-09-11T05:45:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2440376320 bytes 2024-09-11T05:45:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2474123264 bytes 2024-09-11T05:45:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2557768704 bytes 2024-09-11T05:45:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2618027008 bytes 2024-09-11T05:45:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2626539520 bytes 2024-09-11T05:45:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2695972864 bytes 2024-09-11T05:45:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2737071104 bytes 2024-09-11T05:45:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2740867072 bytes 2024-09-11T05:45:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2771962880 bytes 2024-09-11T05:45:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2824404992 bytes 2024-09-11T05:45:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2891395072 bytes 2024-09-11T05:45:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2974595072 bytes 2024-09-11T05:45:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3023468544 bytes 2024-09-11T05:45:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3038008320 bytes 2024-09-11T05:45:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3086070784 bytes 2024-09-11T05:45:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3185646592 bytes 2024-09-11T05:45:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3224187904 bytes 2024-09-11T05:45:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3267622912 bytes 2024-09-11T05:45:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3300426752 bytes 2024-09-11T05:45:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3381074944 bytes 2024-09-11T05:45:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3381677056 bytes 2024-09-11T05:45:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3417219072 bytes 2024-09-11T05:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3517897728 bytes 2024-09-11T05:45:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3552813056 bytes 2024-09-11T05:45:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3597372416 bytes 2024-09-11T05:45:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3635267584 bytes 2024-09-11T05:45:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:45:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=50 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:45:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T05:45:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'images: ' 2024-09-11T05:45:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 2 2024-09-11T05:45:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:45:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=50 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 50 = 50 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2024-09-11T05:45:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:45:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:45:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:45:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T05:45:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'image health:' 2024-09-11T05:45:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 3 2024-09-11T05:45:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:45:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2024-09-11T05:45:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2024-09-11T05:45:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2024-09-11T05:45:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=50 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:45:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:45:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:45:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:45:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T05:45:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'images: ' 2024-09-11T05:45:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 2 2024-09-11T05:45:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=50 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 50 = 50 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:45:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:45:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:45:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2024-09-11T05:45:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:45:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:45:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:45:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T05:45:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'image health:' 2024-09-11T05:45:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 3 2024-09-11T05:45:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:45:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=WARNING 2024-09-11T05:45:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test WARNING = ERROR 2024-09-11T05:45:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test WARNING = OK 2024-09-11T05:45:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:45:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 2 -ne 1 2024-09-11T05:45:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 30 2024-09-11T05:46:02.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.365388393317851) > weight (0.25) 2024-09-11T05:46:02.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8035366105989913) > weight (0.25) 2024-09-11T05:46:02.566 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:46:02.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:46:02.567 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T05:46:02.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:46:02.568+0000 7faa97cad640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40741) UID: 1000 2024-09-11T05:46:09.567 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:46:09.567 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:46:09.568 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:46:09.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:46:09.568 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:46:09.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:46:09.571 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-09-11T05:46:24.590 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.569767366309479) > weight (0.25) 2024-09-11T05:46:24.591 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T05:46:24.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T05:46:24.591 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-11T05:46:24.591 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T05:46:24.591 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:46:24.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:46:24.592+0000 7f00a43cd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39395) UID: 1000 2024-09-11T05:46:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T05:46:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'image health:' 2024-09-11T05:46:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 3 2024-09-11T05:46:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: compare many images' 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:46:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:46:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:46:26 TEST: compare many images 2024-09-11T05:46:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:46:26 TEST: compare many images 2024-09-11T05:46:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 50 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:46:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-11T05:46:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:46:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:46:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:28.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:46:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:46:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:46:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:46:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:46:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:46:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:46:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-11T05:46:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:46:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:46:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:46:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:46:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:46:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_1.status 2024-09-11T05:46:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.status 2024-09-11T05:46:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-11T05:46:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_1.status 2024-09-11T05:46:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-11T05:46:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 86 2024-09-11T05:46:30.008 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b7b58680 2024-09-11T05:46:30.008 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b8bb1d40 2024-09-11T05:46:30.008 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7ba60eb60 2024-09-11T05:46:30.008 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7bcf65a00 2024-09-11T05:46:30.008 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7c108cf70 2024-09-11T05:46:30.009 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:46:30.007+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7c2802270 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.status 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.status 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-11T05:46:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:46:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=86 2024-09-11T05:46:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:46:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T05:46:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=35 2024-09-11T05:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 86 = 35 2024-09-11T05:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:46:30.693 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T05:46:30.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T05:46:30.693 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T05:46:30.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T05:46:30.696 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-11T05:46:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T05:46:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 86 = '' 2024-09-11T05:46:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T05:46:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 86 = '' 2024-09-11T05:46:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:46:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.status 2024-09-11T05:46:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T05:46:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=86 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 86 = 86 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:46:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-11T05:46:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2024-09-11T05:46:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:46:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e86b9b4ae 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e86b9b4ae 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:46:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:19 2024 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:19 2024 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:19 2024 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0081d803-369d-4214-80b0-726959bc08b4 2024-09-11T05:46:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:46:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.export 2024-09-11T05:46:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.export 2024-09-11T05:46:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.export 2024-09-11T05:46:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.export 2024-09-11T05:46:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.export 2024-09-11T05:46:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.export 2024-09-11T05:46:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_1.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_1.export 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-11T05:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-11T05:46:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:46:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:46:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:46:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:46:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:46:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:46:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:46:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:46:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:46:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:46:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:46:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:46:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-11T05:46:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:46:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:46:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:46:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:46:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:46:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:46:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-11T05:46:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_2.status 2024-09-11T05:46:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.status 2024-09-11T05:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-11T05:46:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_2.status 2024-09-11T05:46:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-11T05:46:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 88 2024-09-11T05:46:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.status 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.status 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-11T05:46:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:46:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=88 2024-09-11T05:46:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:46:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T05:46:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=36 2024-09-11T05:46:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 88 = 36 2024-09-11T05:46:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:46:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T05:46:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 88 = '' 2024-09-11T05:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:46:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T05:46:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 88 = '' 2024-09-11T05:46:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:46:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.status 2024-09-11T05:46:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T05:46:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=88 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 88 = 88 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:46:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-11T05:46:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2024-09-11T05:46:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:46:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:46:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:46:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e2e44cf15 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e2e44cf15 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:27 2024 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:27 2024 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:27 2024 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 8de57868-2d26-4418-9611-9fe7ca6aec5d 2024-09-11T05:46:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.export 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.export 2024-09-11T05:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.export 2024-09-11T05:46:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.export 2024-09-11T05:46:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.export 2024-09-11T05:46:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.export 2024-09-11T05:46:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_2.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_2.export 2024-09-11T05:46:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:46:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2024-09-11T05:46:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-11T05:46:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:46:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-11T05:46:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:46:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:46:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:46:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-11T05:46:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:46:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:46:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:46:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:46:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:46:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:46:46.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:46:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:46:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:46:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:46:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:46:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:46:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-11T05:46:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:46:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:46:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:46:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:46:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_3.status 2024-09-11T05:46:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.status 2024-09-11T05:46:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-11T05:46:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_3.status 2024-09-11T05:46:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-11T05:46:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 90 2024-09-11T05:46:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.status 2024-09-11T05:46:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.status 2024-09-11T05:46:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-11T05:46:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=90 2024-09-11T05:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:46:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T05:46:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=37 2024-09-11T05:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 90 = 37 2024-09-11T05:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:46:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T05:46:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 90 = '' 2024-09-11T05:46:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:46:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T05:46:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 90 = '' 2024-09-11T05:46:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:46:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.status 2024-09-11T05:46:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T05:46:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=90 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 90 = 90 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:46:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-11T05:46:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113eb9684104 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113eb9684104 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:42 2024 2024-09-11T05:46:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:42 2024 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:42 2024 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 938fc742-34c2-4279-9e72-cc118f790d8f 2024-09-11T05:46:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.export 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.export 2024-09-11T05:46:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.export 2024-09-11T05:46:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.export 2024-09-11T05:46:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.export 2024-09-11T05:46:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:46:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.export 2024-09-11T05:46:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_3.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_3.export 2024-09-11T05:46:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:46:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:46:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:46:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:46:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:46:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-11T05:46:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:46:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:46:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:46:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:46:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:46:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:46:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:46:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:46:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-11T05:46:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:46:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:46:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:46:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:46:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:46:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-11T05:46:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_4.status 2024-09-11T05:46:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.status 2024-09-11T05:46:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-11T05:46:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_4.status 2024-09-11T05:46:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-11T05:46:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 92 2024-09-11T05:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.status 2024-09-11T05:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.status 2024-09-11T05:46:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-11T05:46:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:46:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=92 2024-09-11T05:46:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:46:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T05:46:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=38 2024-09-11T05:46:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 92 = 38 2024-09-11T05:46:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:46:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T05:46:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 92 = '' 2024-09-11T05:46:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T05:46:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:46:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:46:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:46:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:46:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:46:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:46:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:46:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 92 = '' 2024-09-11T05:46:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:46:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:47:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:47:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.status 2024-09-11T05:47:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T05:47:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=92 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 92 = 92 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:47:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-11T05:47:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e521cf01f 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e521cf01f 2024-09-11T05:47:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:37:49 2024 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:37:49 2024 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:37:49 2024 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 222ccbb3-07f2-4daf-aca6-881dd616be7f 2024-09-11T05:47:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.export 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.export 2024-09-11T05:47:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.export 2024-09-11T05:47:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.export 2024-09-11T05:47:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.export 2024-09-11T05:47:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.export 2024-09-11T05:47:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_4.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_4.export 2024-09-11T05:47:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-11T05:47:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-11T05:47:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T05:47:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_5.status 2024-09-11T05:47:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.status 2024-09-11T05:47:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-11T05:47:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_5.status 2024-09-11T05:47:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-11T05:47:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:05.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-11T05:47:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 94 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.status 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.status 2024-09-11T05:47:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-11T05:47:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=94 2024-09-11T05:47:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T05:47:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=39 2024-09-11T05:47:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 94 = 39 2024-09-11T05:47:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T05:47:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 94 = '' 2024-09-11T05:47:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T05:47:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 94 = '' 2024-09-11T05:47:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.status 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T05:47:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=94 2024-09-11T05:47:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 94 = 94 2024-09-11T05:47:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-11T05:47:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e8b134a69 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e8b134a69 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:03 2024 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:03 2024 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:03 2024 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4109d16e-0d23-49cb-8378-af90cd53b465 2024-09-11T05:47:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-11T05:47:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.export 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.export 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.export 2024-09-11T05:47:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.export 2024-09-11T05:47:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.export 2024-09-11T05:47:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.export 2024-09-11T05:47:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_5.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_5.export 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-11T05:47:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-11T05:47:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-11T05:47:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_6.status 2024-09-11T05:47:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.status 2024-09-11T05:47:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-11T05:47:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_6.status 2024-09-11T05:47:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-11T05:47:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-11T05:47:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 96 2024-09-11T05:47:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.status 2024-09-11T05:47:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.status 2024-09-11T05:47:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-11T05:47:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=96 2024-09-11T05:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T05:47:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=40 2024-09-11T05:47:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 96 = 40 2024-09-11T05:47:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T05:47:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 96 = '' 2024-09-11T05:47:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T05:47:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 96 = '' 2024-09-11T05:47:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.status 2024-09-11T05:47:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T05:47:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=96 2024-09-11T05:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 96 = 96 2024-09-11T05:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-11T05:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e697b73ec 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e697b73ec 2024-09-11T05:47:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:09 2024 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:09 2024 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:09 2024 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fafb8d11-da7d-4858-9912-7044d668ce0c 2024-09-11T05:47:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-11T05:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:47:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.export 2024-09-11T05:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.export 2024-09-11T05:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.export 2024-09-11T05:47:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.export 2024-09-11T05:47:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.export 2024-09-11T05:47:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.export 2024-09-11T05:47:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_6.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_6.export 2024-09-11T05:47:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2024-09-11T05:47:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-11T05:47:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-11T05:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-11T05:47:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-11T05:47:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_7.status 2024-09-11T05:47:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.status 2024-09-11T05:47:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-11T05:47:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_7.status 2024-09-11T05:47:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-11T05:47:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-11T05:47:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 98 2024-09-11T05:47:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.status 2024-09-11T05:47:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.status 2024-09-11T05:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-11T05:47:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=98 2024-09-11T05:47:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T05:47:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=41 2024-09-11T05:47:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 98 = 41 2024-09-11T05:47:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T05:47:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 98 = '' 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T05:47:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 98 = '' 2024-09-11T05:47:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.status 2024-09-11T05:47:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T05:47:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=98 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 98 = 98 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-11T05:47:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2024-09-11T05:47:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113ebf938052 2024-09-11T05:47:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113ebf938052 2024-09-11T05:47:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:22 2024 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:22 2024 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:22 2024 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f6c4c95b-26d9-427d-9fd8-9b7fe2a1b185 2024-09-11T05:47:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.export 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.export 2024-09-11T05:47:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.export 2024-09-11T05:47:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.export 2024-09-11T05:47:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.export 2024-09-11T05:47:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.export 2024-09-11T05:47:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_7.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_7.export 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:31.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-11T05:47:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_8.status 2024-09-11T05:47:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.status 2024-09-11T05:47:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-11T05:47:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_8.status 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-11T05:47:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 100 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.status 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.status 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-11T05:47:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=100 2024-09-11T05:47:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T05:47:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=42 2024-09-11T05:47:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 100 = 42 2024-09-11T05:47:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T05:47:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 100 = '' 2024-09-11T05:47:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T05:47:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 100 = '' 2024-09-11T05:47:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.status 2024-09-11T05:47:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T05:47:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=100 2024-09-11T05:47:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 100 = 100 2024-09-11T05:47:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113ec41d5bb 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113ec41d5bb 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:35 2024 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:35 2024 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:35 2024 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: ab674d81-64fd-4a72-a384-57c22ab78502 2024-09-11T05:47:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.export 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.export 2024-09-11T05:47:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.export 2024-09-11T05:47:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.export 2024-09-11T05:47:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.export 2024-09-11T05:47:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.export 2024-09-11T05:47:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_8.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_8.export 2024-09-11T05:47:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-11T05:47:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-11T05:47:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_9.status 2024-09-11T05:47:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.status 2024-09-11T05:47:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-11T05:47:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_9.status 2024-09-11T05:47:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-11T05:47:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 102 2024-09-11T05:47:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.status 2024-09-11T05:47:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.status 2024-09-11T05:47:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-11T05:47:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=102 2024-09-11T05:47:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:42.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T05:47:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=43 2024-09-11T05:47:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 102 = 43 2024-09-11T05:47:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T05:47:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 102 = '' 2024-09-11T05:47:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T05:47:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:43.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 102 = '' 2024-09-11T05:47:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.status 2024-09-11T05:47:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T05:47:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=102 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 102 = 102 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-11T05:47:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113ef2c67adf 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113ef2c67adf 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:38:47 2024 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:38:47 2024 2024-09-11T05:47:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:38:47 2024 2024-09-11T05:47:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 29554ac5-9f7f-47c5-8ec4-11f432ce8b61 2024-09-11T05:47:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.export 2024-09-11T05:47:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.export 2024-09-11T05:47:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.export 2024-09-11T05:47:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.export 2024-09-11T05:47:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.export 2024-09-11T05:47:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.export 2024-09-11T05:47:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_9.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_9.export 2024-09-11T05:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2024-09-11T05:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-11T05:47:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-11T05:47:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-11T05:47:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-11T05:47:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_10.status 2024-09-11T05:47:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.status 2024-09-11T05:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-11T05:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_10.status 2024-09-11T05:47:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-11T05:47:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-11T05:47:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 104 2024-09-11T05:47:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.status 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.status 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-11T05:47:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=104 2024-09-11T05:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:47:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T05:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=44 2024-09-11T05:47:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 104 = 44 2024-09-11T05:47:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:47:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T05:47:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:51.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2976396755948876) > weight (0.25) 2024-09-11T05:47:51.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8998141149139952) > weight (0.25) 2024-09-11T05:47:51.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2615079021849718) > weight (0.25) 2024-09-11T05:47:51.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.39981691857527735) > weight (0.25) 2024-09-11T05:47:51.572 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-11T05:47:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 104 = '' 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T05:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:47:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:47:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:47:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:47:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 104 = '' 2024-09-11T05:47:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.status 2024-09-11T05:47:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T05:47:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:47:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=104 2024-09-11T05:47:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 104 = 104 2024-09-11T05:47:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:47:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e80c3d52 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e80c3d52 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:02 2024 2024-09-11T05:47:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:02 2024 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:02 2024 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 417b48dd-e251-40f1-872a-99ad8bf806db 2024-09-11T05:47:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:47:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.export 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.export 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.export 2024-09-11T05:47:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.export 2024-09-11T05:47:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.export 2024-09-11T05:47:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:47:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.export 2024-09-11T05:47:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_10.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_10.export 2024-09-11T05:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:47:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:47:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:47:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-11T05:47:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:47:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:47:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:47:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:47:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:47:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:47:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:47:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:47:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:47:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:47:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:47:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:47:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:47:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-11T05:47:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:47:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:47:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:47:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:47:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_11.status 2024-09-11T05:47:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.status 2024-09-11T05:47:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-11T05:47:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_11.status 2024-09-11T05:47:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:47:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:47:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-11T05:47:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-11T05:47:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 106 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.status 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.status 2024-09-11T05:47:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-11T05:47:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:47:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:47:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=106 2024-09-11T05:47:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:47:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:47:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T05:48:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=45 2024-09-11T05:48:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 106 = 45 2024-09-11T05:48:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T05:48:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 106 = '' 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T05:48:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 106 = '' 2024-09-11T05:48:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.status 2024-09-11T05:48:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T05:48:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=106 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 106 = 106 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_11': 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e4efca87f 2024-09-11T05:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e4efca87f 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:16 2024 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:16 2024 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:16 2024 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f5febb24-4281-4e75-9b77-08bba442bf19 2024-09-11T05:48:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.export 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.export 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.export 2024-09-11T05:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.export 2024-09-11T05:48:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.export 2024-09-11T05:48:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.export 2024-09-11T05:48:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_11.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_11.export 2024-09-11T05:48:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:07.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-11T05:48:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_12.status 2024-09-11T05:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.status 2024-09-11T05:48:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-11T05:48:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_12.status 2024-09-11T05:48:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-11T05:48:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 108 2024-09-11T05:48:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.status 2024-09-11T05:48:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.status 2024-09-11T05:48:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-11T05:48:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=108 2024-09-11T05:48:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T05:48:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=46 2024-09-11T05:48:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 108 = 46 2024-09-11T05:48:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T05:48:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 108 = '' 2024-09-11T05:48:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T05:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 108 = '' 2024-09-11T05:48:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:10.697 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.504935638982367) > weight (0.25) 2024-09-11T05:48:10.697 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T05:48:10.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T05:48:10.697 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:48:10.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:48:10.702+0000 7f3129871640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 42153) UID: 1000 2024-09-11T05:48:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.status 2024-09-11T05:48:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T05:48:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=108 2024-09-11T05:48:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 108 = 108 2024-09-11T05:48:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-11T05:48:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_12': 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e52c9a864 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e52c9a864 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:30 2024 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:30 2024 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:30 2024 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b16094f1-b9fc-4dcb-bceb-0331cdd63e45 2024-09-11T05:48:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-11T05:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.export 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.export 2024-09-11T05:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.export 2024-09-11T05:48:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.export 2024-09-11T05:48:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.export 2024-09-11T05:48:13.698 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T05:48:13.698 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:48:13.699 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T05:48:13.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T05:48:13.699 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T05:48:13.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T05:48:13.702 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-11T05:48:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.export 2024-09-11T05:48:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_12.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_12.export 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-11T05:48:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-11T05:48:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-11T05:48:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-11T05:48:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_13.status 2024-09-11T05:48:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.status 2024-09-11T05:48:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-11T05:48:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_13.status 2024-09-11T05:48:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-11T05:48:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 110 2024-09-11T05:48:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.status 2024-09-11T05:48:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.status 2024-09-11T05:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-11T05:48:17.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=110 2024-09-11T05:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T05:48:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=47 2024-09-11T05:48:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 110 = 47 2024-09-11T05:48:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T05:48:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 110 = '' 2024-09-11T05:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T05:48:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 110 = '' 2024-09-11T05:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.status 2024-09-11T05:48:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T05:48:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=110 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 110 = 110 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-11T05:48:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_13': 2024-09-11T05:48:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e5f3504bf 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e5f3504bf 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:40 2024 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:40 2024 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:40 2024 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a98d5b12-6ee4-49e3-b583-ddcaf0392d73 2024-09-11T05:48:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.export 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.export 2024-09-11T05:48:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.export 2024-09-11T05:48:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.export 2024-09-11T05:48:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.export 2024-09-11T05:48:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.export 2024-09-11T05:48:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_13.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_13.export 2024-09-11T05:48:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2024-09-11T05:48:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-11T05:48:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-11T05:48:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-11T05:48:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-11T05:48:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_14.status 2024-09-11T05:48:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.status 2024-09-11T05:48:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-11T05:48:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_14.status 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-11T05:48:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 112 2024-09-11T05:48:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.status 2024-09-11T05:48:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.status 2024-09-11T05:48:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-11T05:48:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=112 2024-09-11T05:48:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T05:48:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=48 2024-09-11T05:48:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 112 = 48 2024-09-11T05:48:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T05:48:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 112 = '' 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T05:48:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 112 = '' 2024-09-11T05:48:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.status 2024-09-11T05:48:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T05:48:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=112 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 112 = 112 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-11T05:48:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_14': 2024-09-11T05:48:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113eb6122917 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113eb6122917 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:50 2024 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:50 2024 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:50 2024 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5e3847d7-8152-4a70-94e2-5fa8963bddcf 2024-09-11T05:48:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.export 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.export 2024-09-11T05:48:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.export 2024-09-11T05:48:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.export 2024-09-11T05:48:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.export 2024-09-11T05:48:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.export 2024-09-11T05:48:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_14.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_14.export 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-11T05:48:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-11T05:48:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-11T05:48:35.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_15.status 2024-09-11T05:48:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.status 2024-09-11T05:48:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-11T05:48:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_15.status 2024-09-11T05:48:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-11T05:48:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-11T05:48:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 114 2024-09-11T05:48:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.status 2024-09-11T05:48:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.status 2024-09-11T05:48:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-11T05:48:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=114 2024-09-11T05:48:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T05:48:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=49 2024-09-11T05:48:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 114 = 49 2024-09-11T05:48:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T05:48:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 114 = '' 2024-09-11T05:48:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T05:48:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 114 = '' 2024-09-11T05:48:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.status 2024-09-11T05:48:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T05:48:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=114 2024-09-11T05:48:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 114 = 114 2024-09-11T05:48:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-11T05:48:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_15': 2024-09-11T05:48:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e70dc5c35 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e70dc5c35 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:39:57 2024 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:39:57 2024 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:39:57 2024 2024-09-11T05:48:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 61ff442c-b39c-42e8-bcaf-1b8c56417011 2024-09-11T05:48:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.export 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.export 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.export 2024-09-11T05:48:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.export 2024-09-11T05:48:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.export 2024-09-11T05:48:40.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.export 2024-09-11T05:48:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_15.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_15.export 2024-09-11T05:48:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2024-09-11T05:48:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-11T05:48:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-11T05:48:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_16.status 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.status 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-11T05:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_16.status 2024-09-11T05:48:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-11T05:48:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 116 2024-09-11T05:48:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.status 2024-09-11T05:48:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.status 2024-09-11T05:48:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-11T05:48:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=116 2024-09-11T05:48:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T05:48:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=50 2024-09-11T05:48:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 116 = 50 2024-09-11T05:48:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T05:48:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 116 = '' 2024-09-11T05:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T05:48:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 116 = '' 2024-09-11T05:48:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.status 2024-09-11T05:48:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T05:48:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=116 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 116 = 116 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_16': 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e3f971c1e 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e3f971c1e 2024-09-11T05:48:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:11 2024 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:11 2024 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:11 2024 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e19e65a3-67e6-4b67-bbe6-97b140e081a0 2024-09-11T05:48:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.export 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.export 2024-09-11T05:48:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.export 2024-09-11T05:48:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.export 2024-09-11T05:48:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.export 2024-09-11T05:48:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.export 2024-09-11T05:48:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_16.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_16.export 2024-09-11T05:48:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:48:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:48:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:48:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:48:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:48:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:48:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:48:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:48:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:48:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:48:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:48:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:48:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:48:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:48:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-11T05:48:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:48:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:48:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:48:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:48:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-11T05:48:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_17.status 2024-09-11T05:48:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.status 2024-09-11T05:48:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-11T05:48:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_17.status 2024-09-11T05:48:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:48:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-11T05:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-11T05:48:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 118 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.status 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.status 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-11T05:48:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:48:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=118 2024-09-11T05:48:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:48:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T05:48:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=51 2024-09-11T05:48:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 118 = 51 2024-09-11T05:48:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:48:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T05:48:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 118 = '' 2024-09-11T05:48:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:48:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T05:48:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:48:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:48:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 118 = '' 2024-09-11T05:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:48:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.status 2024-09-11T05:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T05:48:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:48:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:48:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=118 2024-09-11T05:48:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 118 = 118 2024-09-11T05:48:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:48:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:48:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-11T05:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_17': 2024-09-11T05:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:48:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113eace5a75e 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113eace5a75e 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:24 2024 2024-09-11T05:48:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:24 2024 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:24 2024 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e 2024-09-11T05:48:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:48:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-11T05:48:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:48:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:48:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.export 2024-09-11T05:48:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.export 2024-09-11T05:48:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.export 2024-09-11T05:48:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.export 2024-09-11T05:48:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.export 2024-09-11T05:48:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:48:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.export 2024-09-11T05:48:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_17.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_17.export 2024-09-11T05:48:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:48:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:48:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-11T05:49:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-11T05:49:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-11T05:49:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-11T05:49:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_18.status 2024-09-11T05:49:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.status 2024-09-11T05:49:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-11T05:49:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_18.status 2024-09-11T05:49:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-11T05:49:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-11T05:49:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 120 2024-09-11T05:49:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.status 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.status 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-11T05:49:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=120 2024-09-11T05:49:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T05:49:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=52 2024-09-11T05:49:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 120 = 52 2024-09-11T05:49:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T05:49:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 120 = '' 2024-09-11T05:49:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T05:49:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 120 = '' 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:06.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.status 2024-09-11T05:49:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T05:49:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=120 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 120 = 120 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-11T05:49:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_18': 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e27b29b3b 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e27b29b3b 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:34 2024 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:34 2024 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:34 2024 2024-09-11T05:49:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7f727691-cb67-470b-9ad0-990f37a8f164 2024-09-11T05:49:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-11T05:49:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.export 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.export 2024-09-11T05:49:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.export 2024-09-11T05:49:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.export 2024-09-11T05:49:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.export 2024-09-11T05:49:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.export 2024-09-11T05:49:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_18.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_18.export 2024-09-11T05:49:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-11T05:49:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-11T05:49:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-11T05:49:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_19.status 2024-09-11T05:49:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.status 2024-09-11T05:49:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-11T05:49:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_19.status 2024-09-11T05:49:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-11T05:49:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-11T05:49:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 122 2024-09-11T05:49:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.status 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.status 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-11T05:49:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=122 2024-09-11T05:49:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T05:49:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=53 2024-09-11T05:49:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 122 = 53 2024-09-11T05:49:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T05:49:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 122 = '' 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T05:49:13.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 122 = '' 2024-09-11T05:49:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.status 2024-09-11T05:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T05:49:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=122 2024-09-11T05:49:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 122 = 122 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-11T05:49:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_19': 2024-09-11T05:49:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e3171dc22 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e3171dc22 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:43 2024 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:43 2024 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:43 2024 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 18a350ae-4c11-4211-aa64-b6afa6f108ba 2024-09-11T05:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.export 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.export 2024-09-11T05:49:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.export 2024-09-11T05:49:16.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.export 2024-09-11T05:49:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.export 2024-09-11T05:49:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.export 2024-09-11T05:49:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_19.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_19.export 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:18.573 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4469025474477276) > weight (0.25) 2024-09-11T05:49:18.573 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9226492952187416) > weight (0.25) 2024-09-11T05:49:18.573 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4273869682640171) > weight (0.25) 2024-09-11T05:49:18.574 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-11T05:49:18.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-11T05:49:18.574 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T05:49:18.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T05:49:18.576+0000 7f603d0a1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 28918) UID: 1000 2024-09-11T05:49:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-11T05:49:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-11T05:49:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_20.status 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.status 2024-09-11T05:49:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-11T05:49:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_20.status 2024-09-11T05:49:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-11T05:49:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 124 2024-09-11T05:49:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.status 2024-09-11T05:49:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.status 2024-09-11T05:49:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-11T05:49:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=124 2024-09-11T05:49:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T05:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=54 2024-09-11T05:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 124 = 54 2024-09-11T05:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T05:49:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 124 = '' 2024-09-11T05:49:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T05:49:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:22.575 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-11T05:49:22.575 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:49:22.575 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-11T05:49:22.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T05:49:22.576 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T05:49:22.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T05:49:22.578 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-11T05:49:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 124 = '' 2024-09-11T05:49:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.status 2024-09-11T05:49:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T05:49:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=124 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 124 = 124 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-11T05:49:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_20': 2024-09-11T05:49:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 113e3e5cdad4 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.113e3e5cdad4 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:40:52 2024 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:40:52 2024 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:40:52 2024 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4607f120-f3a3-4ead-9c81-2256f6efd774 2024-09-11T05:49:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.export 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.export 2024-09-11T05:49:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.export 2024-09-11T05:49:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.export 2024-09-11T05:49:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.export 2024-09-11T05:49:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.export 2024-09-11T05:49:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_20.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_20.export 2024-09-11T05:49:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-11T05:49:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-11T05:49:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:29.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_21.status 2024-09-11T05:49:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.status 2024-09-11T05:49:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-11T05:49:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_21.status 2024-09-11T05:49:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-11T05:49:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 126 2024-09-11T05:49:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.status 2024-09-11T05:49:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.status 2024-09-11T05:49:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-11T05:49:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=126 2024-09-11T05:49:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T05:49:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=55 2024-09-11T05:49:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 126 = 55 2024-09-11T05:49:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T05:49:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 126 = '' 2024-09-11T05:49:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T05:49:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 126 = '' 2024-09-11T05:49:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.status 2024-09-11T05:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T05:49:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:33.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=126 2024-09-11T05:49:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 126 = 126 2024-09-11T05:49:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_21': 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7a0820b76 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7a0820b76 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:06 2024 2024-09-11T05:49:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:06 2024 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:06 2024 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d6371132-753a-4a4f-8524-8425689888b8 2024-09-11T05:49:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.export 2024-09-11T05:49:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.export 2024-09-11T05:49:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.export 2024-09-11T05:49:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.export 2024-09-11T05:49:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.export 2024-09-11T05:49:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.export 2024-09-11T05:49:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_21.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_21.export 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-11T05:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-11T05:49:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-11T05:49:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_22.status 2024-09-11T05:49:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.status 2024-09-11T05:49:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-11T05:49:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_22.status 2024-09-11T05:49:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-11T05:49:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 128 2024-09-11T05:49:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.status 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.status 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-11T05:49:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=128 2024-09-11T05:49:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T05:49:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=56 2024-09-11T05:49:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 128 = 56 2024-09-11T05:49:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T05:49:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 128 = '' 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T05:49:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 128 = '' 2024-09-11T05:49:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.status 2024-09-11T05:49:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T05:49:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=128 2024-09-11T05:49:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 128 = 128 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_22': 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7f98d0b73 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7f98d0b73 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:17 2024 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:17 2024 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:17 2024 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: db729295-01a4-4d5f-aec1-17d8b32aefee 2024-09-11T05:49:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.export 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.export 2024-09-11T05:49:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.export 2024-09-11T05:49:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.export 2024-09-11T05:49:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.export 2024-09-11T05:49:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.export 2024-09-11T05:49:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_22.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_22.export 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:46.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:46.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-11T05:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-11T05:49:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_23.status 2024-09-11T05:49:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.status 2024-09-11T05:49:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-11T05:49:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_23.status 2024-09-11T05:49:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-11T05:49:48.003 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2024-09-11T05:49:48.004+0000 7f3afe201640 -1 reset not still connected to 0x565207e368f0 2024-09-11T05:49:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 130 2024-09-11T05:49:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.status 2024-09-11T05:49:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.status 2024-09-11T05:49:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-11T05:49:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=130 2024-09-11T05:49:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T05:49:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=57 2024-09-11T05:49:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 130 = 57 2024-09-11T05:49:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T05:49:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 130 = '' 2024-09-11T05:49:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T05:49:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 130 = '' 2024-09-11T05:49:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.status 2024-09-11T05:49:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T05:49:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=130 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 130 = 130 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-11T05:49:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_23': 2024-09-11T05:49:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:49:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:49:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c782d575ad 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c782d575ad 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:31 2024 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:31 2024 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:31 2024 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:49:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7 2024-09-11T05:49:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.export 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.export 2024-09-11T05:49:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.export 2024-09-11T05:49:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.export 2024-09-11T05:49:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.export 2024-09-11T05:49:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:49:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.export 2024-09-11T05:49:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_23.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_23.export 2024-09-11T05:49:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:49:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2024-09-11T05:49:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-11T05:49:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:49:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:49:53.703 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:49:53.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:49:53.704 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T05:49:53.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:49:53.704+0000 7f2d2ff6d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 41277) UID: 1000 2024-09-11T05:49:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-11T05:49:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:49:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:49:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:49:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:49:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:49:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:49:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:49:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:49:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-11T05:49:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:49:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:49:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:49:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:49:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:49:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:49:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:49:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:49:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:49:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-11T05:49:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:49:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:49:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:49:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:49:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:49:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_24.status 2024-09-11T05:49:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.status 2024-09-11T05:49:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-11T05:49:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_24.status 2024-09-11T05:49:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:49:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-11T05:49:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-11T05:49:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 132 2024-09-11T05:49:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.status 2024-09-11T05:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.status 2024-09-11T05:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-11T05:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:49:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=132 2024-09-11T05:49:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:49:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T05:49:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=58 2024-09-11T05:49:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 132 = 58 2024-09-11T05:49:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T05:49:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 132 = '' 2024-09-11T05:49:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:49:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:49:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T05:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:49:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:49:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:49:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:49:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:49:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:49:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:49:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 132 = '' 2024-09-11T05:49:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:49:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:50:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:50:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.status 2024-09-11T05:50:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T05:50:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=132 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 132 = 132 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-11T05:50:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_24': 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7be2bc01f 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7be2bc01f 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:42 2024 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:42 2024 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:42 2024 2024-09-11T05:50:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 73f51a71-2b0c-4c23-b0d3-ddf1f35a1a67 2024-09-11T05:50:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.export 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.export 2024-09-11T05:50:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.export 2024-09-11T05:50:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.export 2024-09-11T05:50:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.export 2024-09-11T05:50:01.704 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:50:01.704 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:50:01.705 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:50:01.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:50:01.705 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:50:01.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:50:01.708 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-11T05:50:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.export 2024-09-11T05:50:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_24.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_24.export 2024-09-11T05:50:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-11T05:50:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-11T05:50:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-11T05:50:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_25.status 2024-09-11T05:50:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.status 2024-09-11T05:50:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-11T05:50:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_25.status 2024-09-11T05:50:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-11T05:50:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-11T05:50:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 134 2024-09-11T05:50:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.status 2024-09-11T05:50:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.status 2024-09-11T05:50:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-11T05:50:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=134 2024-09-11T05:50:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T05:50:06.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=59 2024-09-11T05:50:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 134 = 59 2024-09-11T05:50:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T05:50:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 134 = '' 2024-09-11T05:50:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T05:50:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 134 = '' 2024-09-11T05:50:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.status 2024-09-11T05:50:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T05:50:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=134 2024-09-11T05:50:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 134 = 134 2024-09-11T05:50:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-11T05:50:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_25': 2024-09-11T05:50:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7b02ace82 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7b02ace82 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:41:57 2024 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:41:57 2024 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:41:57 2024 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a87873a5-1cad-45f5-a638-56e77deab3e3 2024-09-11T05:50:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-11T05:50:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:50:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.export 2024-09-11T05:50:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.export 2024-09-11T05:50:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.export 2024-09-11T05:50:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.export 2024-09-11T05:50:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.export 2024-09-11T05:50:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.export 2024-09-11T05:50:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_25.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_25.export 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-11T05:50:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-11T05:50:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_26.status 2024-09-11T05:50:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.status 2024-09-11T05:50:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-11T05:50:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_26.status 2024-09-11T05:50:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-11T05:50:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 136 2024-09-11T05:50:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.status 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.status 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-11T05:50:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=136 2024-09-11T05:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T05:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=60 2024-09-11T05:50:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 136 = 60 2024-09-11T05:50:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T05:50:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 136 = '' 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T05:50:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 136 = '' 2024-09-11T05:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.status 2024-09-11T05:50:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T05:50:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=136 2024-09-11T05:50:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 136 = 136 2024-09-11T05:50:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-11T05:50:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_26': 2024-09-11T05:50:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7ddea3e11 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7ddea3e11 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:04 2024 2024-09-11T05:50:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:04 2024 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:04 2024 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 95e24eb9-a635-4d8b-b286-5bc01f045ba1 2024-09-11T05:50:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.export 2024-09-11T05:50:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.export 2024-09-11T05:50:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.export 2024-09-11T05:50:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.export 2024-09-11T05:50:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.export 2024-09-11T05:50:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.export 2024-09-11T05:50:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_26.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_26.export 2024-09-11T05:50:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-11T05:50:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-11T05:50:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-11T05:50:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_27.status 2024-09-11T05:50:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.status 2024-09-11T05:50:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-11T05:50:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_27.status 2024-09-11T05:50:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-11T05:50:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-11T05:50:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 138 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.status 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.status 2024-09-11T05:50:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-11T05:50:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=138 2024-09-11T05:50:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T05:50:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=61 2024-09-11T05:50:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 138 = 61 2024-09-11T05:50:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T05:50:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=61 2024-09-11T05:50:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 138 = 61 2024-09-11T05:50:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T05:50:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 138 = '' 2024-09-11T05:50:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.status 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T05:50:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=138 2024-09-11T05:50:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 138 = 138 2024-09-11T05:50:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_27': 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c717372ab9 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c717372ab9 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:16 2024 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:16 2024 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:16 2024 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1641c3d7-e0a2-407c-a874-91a1ef797a0c 2024-09-11T05:50:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-11T05:50:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:50:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.export 2024-09-11T05:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.export 2024-09-11T05:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.export 2024-09-11T05:50:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.export 2024-09-11T05:50:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.export 2024-09-11T05:50:28.579 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.46105754199307913) > weight (0.25) 2024-09-11T05:50:28.580 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7186525243206053) > weight (0.25) 2024-09-11T05:50:28.580 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.36132207292577956) > weight (0.25) 2024-09-11T05:50:28.580 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6763707131151049) > weight (0.25) 2024-09-11T05:50:28.580 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-11T05:50:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.export 2024-09-11T05:50:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_27.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_27.export 2024-09-11T05:50:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:31.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-11T05:50:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-11T05:50:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_28.status 2024-09-11T05:50:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.status 2024-09-11T05:50:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-11T05:50:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_28.status 2024-09-11T05:50:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-11T05:50:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-11T05:50:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 140 2024-09-11T05:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.status 2024-09-11T05:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.status 2024-09-11T05:50:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-11T05:50:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=140 2024-09-11T05:50:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T05:50:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=62 2024-09-11T05:50:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 140 = 62 2024-09-11T05:50:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T05:50:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 140 = '' 2024-09-11T05:50:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T05:50:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 140 = '' 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.status 2024-09-11T05:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T05:50:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=140 2024-09-11T05:50:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 140 = 140 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_28': 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c73e36ebe2 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c73e36ebe2 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:23 2024 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:23 2024 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:23 2024 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 240aff3e-f4c4-4e78-89e9-f002bb8ec31b 2024-09-11T05:50:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.export 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.export 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.export 2024-09-11T05:50:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.export 2024-09-11T05:50:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.export 2024-09-11T05:50:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.export 2024-09-11T05:50:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_28.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_28.export 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-11T05:50:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-11T05:50:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-11T05:50:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_29.status 2024-09-11T05:50:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.status 2024-09-11T05:50:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-11T05:50:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_29.status 2024-09-11T05:50:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-11T05:50:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-11T05:50:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 142 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.status 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.status 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-11T05:50:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=142 2024-09-11T05:50:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T05:50:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=63 2024-09-11T05:50:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 142 = 63 2024-09-11T05:50:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T05:50:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 142 = '' 2024-09-11T05:50:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:43.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T05:50:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 142 = '' 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.status 2024-09-11T05:50:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T05:50:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=142 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 142 = 142 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-11T05:50:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_29': 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c794e0d6de 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c794e0d6de 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:32 2024 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:32 2024 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:32 2024 2024-09-11T05:50:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2c6de312-17ab-498c-82cf-a905326cbb84 2024-09-11T05:50:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-11T05:50:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.export 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.export 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.export 2024-09-11T05:50:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.export 2024-09-11T05:50:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.export 2024-09-11T05:50:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.export 2024-09-11T05:50:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_29.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_29.export 2024-09-11T05:50:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-11T05:50:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-11T05:50:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-11T05:50:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-11T05:50:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_30.status 2024-09-11T05:50:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.status 2024-09-11T05:50:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-11T05:50:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_30.status 2024-09-11T05:50:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-11T05:50:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 144 2024-09-11T05:50:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.status 2024-09-11T05:50:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.status 2024-09-11T05:50:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-11T05:50:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:50:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=144 2024-09-11T05:50:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:50:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T05:50:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=64 2024-09-11T05:50:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 144 = 64 2024-09-11T05:50:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T05:50:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 144 = '' 2024-09-11T05:50:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:50:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T05:50:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:50:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:50:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:50:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:50:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:50:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 144 = '' 2024-09-11T05:50:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:50:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.status 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T05:50:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:50:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=144 2024-09-11T05:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 144 = 144 2024-09-11T05:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-11T05:50:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:50:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-11T05:50:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_30': 2024-09-11T05:50:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:50:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7e6956145 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7e6956145 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:40 2024 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:40 2024 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:40 2024 2024-09-11T05:50:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:50:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:50:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:50:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1c1a7c61-6e2b-48cc-80d9-2dcd7ffe860a 2024-09-11T05:50:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:50:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-11T05:50:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.export 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.export 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.export 2024-09-11T05:50:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.export 2024-09-11T05:50:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.export 2024-09-11T05:50:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:50:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.export 2024-09-11T05:50:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_30.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_30.export 2024-09-11T05:50:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:50:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:50:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:50:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:50:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:50:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:50:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:50:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:50:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:50:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:50:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-11T05:50:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:50:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:50:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:50:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:50:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:50:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-11T05:50:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_31.status 2024-09-11T05:50:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.status 2024-09-11T05:50:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-11T05:50:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_31.status 2024-09-11T05:50:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:50:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-11T05:51:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 146 2024-09-11T05:51:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.status 2024-09-11T05:51:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.status 2024-09-11T05:51:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-11T05:51:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=146 2024-09-11T05:51:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T05:51:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=65 2024-09-11T05:51:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 146 = 65 2024-09-11T05:51:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T05:51:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 146 = '' 2024-09-11T05:51:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T05:51:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 146 = '' 2024-09-11T05:51:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.status 2024-09-11T05:51:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T05:51:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=146 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 146 = 146 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-11T05:51:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_31': 2024-09-11T05:51:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7d2e097f9 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7d2e097f9 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:49 2024 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:49 2024 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:49 2024 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 57412544-618d-4457-b99f-bd8f708b1bd6 2024-09-11T05:51:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-11T05:51:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:51:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.export 2024-09-11T05:51:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.export 2024-09-11T05:51:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.export 2024-09-11T05:51:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.export 2024-09-11T05:51:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.export 2024-09-11T05:51:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.export 2024-09-11T05:51:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_31.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_31.export 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:08.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-11T05:51:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-11T05:51:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-11T05:51:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_32.status 2024-09-11T05:51:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.status 2024-09-11T05:51:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-11T05:51:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_32.status 2024-09-11T05:51:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-11T05:51:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 148 2024-09-11T05:51:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.status 2024-09-11T05:51:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.status 2024-09-11T05:51:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-11T05:51:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=148 2024-09-11T05:51:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T05:51:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=66 2024-09-11T05:51:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 148 = 66 2024-09-11T05:51:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T05:51:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 148 = '' 2024-09-11T05:51:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T05:51:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 148 = '' 2024-09-11T05:51:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.status 2024-09-11T05:51:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T05:51:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=148 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 148 = 148 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_32': 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c7bd4aca94 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c7bd4aca94 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:42:56 2024 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:42:56 2024 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:42:56 2024 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: a4730635-d6c8-461d-89b9-ef37acfa7eec 2024-09-11T05:51:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.export 2024-09-11T05:51:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.export 2024-09-11T05:51:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.export 2024-09-11T05:51:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.export 2024-09-11T05:51:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.export 2024-09-11T05:51:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.export 2024-09-11T05:51:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_32.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_32.export 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-11T05:51:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-11T05:51:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-11T05:51:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_33.status 2024-09-11T05:51:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.status 2024-09-11T05:51:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-11T05:51:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_33.status 2024-09-11T05:51:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-11T05:51:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-11T05:51:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 150 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.status 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.status 2024-09-11T05:51:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-11T05:51:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=150 2024-09-11T05:51:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T05:51:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=67 2024-09-11T05:51:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 150 = 67 2024-09-11T05:51:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T05:51:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 150 = '' 2024-09-11T05:51:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T05:51:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 150 = '' 2024-09-11T05:51:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.status 2024-09-11T05:51:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T05:51:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=150 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 150 = 150 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-11T05:51:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_33': 2024-09-11T05:51:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b157a8a75 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b157a8a75 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:11 2024 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:11 2024 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:11 2024 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fe60b457-0a30-4f36-a30e-adc07ffaf32e 2024-09-11T05:51:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-11T05:51:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.export 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.export 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.export 2024-09-11T05:51:22.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.export 2024-09-11T05:51:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.export 2024-09-11T05:51:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.export 2024-09-11T05:51:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_33.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_33.export 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-11T05:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-11T05:51:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-11T05:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_34.status 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.status 2024-09-11T05:51:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-11T05:51:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_34.status 2024-09-11T05:51:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-11T05:51:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-11T05:51:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 152 2024-09-11T05:51:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.status 2024-09-11T05:51:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.status 2024-09-11T05:51:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-11T05:51:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=152 2024-09-11T05:51:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T05:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=68 2024-09-11T05:51:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 152 = 68 2024-09-11T05:51:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T05:51:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 152 = '' 2024-09-11T05:51:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T05:51:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 152 = '' 2024-09-11T05:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:29.580 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.619397125729716) > weight (0.25) 2024-09-11T05:51:29.581 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4327807566630607) > weight (0.25) 2024-09-11T05:51:29.581 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T05:51:29.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T05:51:29.582 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:51:29.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T05:51:29.582+0000 7f7cb913a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 41932) UID: 1000 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.status 2024-09-11T05:51:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T05:51:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=152 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 152 = 152 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-11T05:51:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_34': 2024-09-11T05:51:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b8feff14d 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b8feff14d 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:22 2024 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:22 2024 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:22 2024 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 82196961-27ae-4fd0-a78e-c8801a4e74b1 2024-09-11T05:51:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-11T05:51:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:51:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.export 2024-09-11T05:51:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.export 2024-09-11T05:51:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.export 2024-09-11T05:51:31.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.export 2024-09-11T05:51:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.export 2024-09-11T05:51:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.export 2024-09-11T05:51:32.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_34.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_34.export 2024-09-11T05:51:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-11T05:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-11T05:51:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-11T05:51:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_35.status 2024-09-11T05:51:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.status 2024-09-11T05:51:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-11T05:51:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_35.status 2024-09-11T05:51:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-11T05:51:35.582 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T05:51:35.582 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:51:35.583 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T05:51:35.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T05:51:35.583 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T05:51:35.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T05:51:35.586 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-11T05:51:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 154 2024-09-11T05:51:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.status 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.status 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-11T05:51:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=154 2024-09-11T05:51:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T05:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=69 2024-09-11T05:51:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 154 = 69 2024-09-11T05:51:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T05:51:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 154 = '' 2024-09-11T05:51:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T05:51:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 154 = '' 2024-09-11T05:51:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.status 2024-09-11T05:51:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T05:51:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=154 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 154 = 154 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_35': 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b8163067c 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b8163067c 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:33 2024 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:33 2024 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:33 2024 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4c30e781-f967-4701-9b49-4258268b118a 2024-09-11T05:51:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.export 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.export 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.export 2024-09-11T05:51:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.export 2024-09-11T05:51:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.export 2024-09-11T05:51:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.export 2024-09-11T05:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_35.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_35.export 2024-09-11T05:51:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2024-09-11T05:51:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-11T05:51:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-11T05:51:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-11T05:51:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-11T05:51:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_36.status 2024-09-11T05:51:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.status 2024-09-11T05:51:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-11T05:51:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_36.status 2024-09-11T05:51:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-11T05:51:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 156 2024-09-11T05:51:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.status 2024-09-11T05:51:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.status 2024-09-11T05:51:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-11T05:51:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=156 2024-09-11T05:51:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T05:51:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=70 2024-09-11T05:51:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 156 = 70 2024-09-11T05:51:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T05:51:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 156 = '' 2024-09-11T05:51:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T05:51:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 156 = '' 2024-09-11T05:51:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.status 2024-09-11T05:51:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T05:51:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=156 2024-09-11T05:51:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 156 = 156 2024-09-11T05:51:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-11T05:51:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-11T05:51:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_36': 2024-09-11T05:51:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b9697e8f 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b9697e8f 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:47 2024 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:47 2024 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:47 2024 2024-09-11T05:51:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9c4bb36c-7150-40d3-a228-8a4295768e81 2024-09-11T05:51:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.export 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.export 2024-09-11T05:51:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.export 2024-09-11T05:51:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.export 2024-09-11T05:51:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.export 2024-09-11T05:51:49.708 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6743713836715545) > weight (0.25) 2024-09-11T05:51:49.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7014236789160244) > weight (0.25) 2024-09-11T05:51:49.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9441709469579852) > weight (0.25) 2024-09-11T05:51:49.709 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8941353694800603) > weight (0.25) 2024-09-11T05:51:49.709 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-09-11T05:51:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.export 2024-09-11T05:51:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_36.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_36.export 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:51:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:51:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:51:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:51:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:51:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:51:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:51:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:51:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:51:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:51:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:51:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:51:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-11T05:51:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:51:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:51:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:51:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:51:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_37.status 2024-09-11T05:51:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.status 2024-09-11T05:51:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-11T05:51:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_37.status 2024-09-11T05:51:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-11T05:51:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 158 2024-09-11T05:51:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.status 2024-09-11T05:51:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.status 2024-09-11T05:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-11T05:51:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:51:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=158 2024-09-11T05:51:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:51:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T05:51:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=71 2024-09-11T05:51:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 158 = 71 2024-09-11T05:51:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:51:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T05:51:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 158 = '' 2024-09-11T05:51:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T05:51:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:51:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:51:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:51:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:51:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:51:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 158 = '' 2024-09-11T05:51:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:51:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.status 2024-09-11T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T05:51:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:51:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:51:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=158 2024-09-11T05:51:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 158 = 158 2024-09-11T05:51:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:51:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:51:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_37': 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b18f2b90a 2024-09-11T05:51:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b18f2b90a 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:43:57 2024 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:43:57 2024 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:43:57 2024 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f496dce6-4d07-4bf7-9ba7-d6c8866c01dc 2024-09-11T05:51:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:51:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.export 2024-09-11T05:51:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.export 2024-09-11T05:51:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.export 2024-09-11T05:51:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.export 2024-09-11T05:51:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.export 2024-09-11T05:51:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.export 2024-09-11T05:51:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_37.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_37.export 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:51:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-11T05:52:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-11T05:52:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-11T05:52:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-11T05:52:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_38.status 2024-09-11T05:52:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.status 2024-09-11T05:52:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-11T05:52:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_38.status 2024-09-11T05:52:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-11T05:52:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-11T05:52:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 160 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.status 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.status 2024-09-11T05:52:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-11T05:52:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=160 2024-09-11T05:52:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T05:52:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=72 2024-09-11T05:52:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 160 = 72 2024-09-11T05:52:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T05:52:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 160 = '' 2024-09-11T05:52:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T05:52:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 160 = '' 2024-09-11T05:52:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.status 2024-09-11T05:52:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T05:52:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=160 2024-09-11T05:52:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 160 = 160 2024-09-11T05:52:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-11T05:52:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_38': 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b878e44b2 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b878e44b2 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:06 2024 2024-09-11T05:52:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:06 2024 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:06 2024 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 62c8df86-31da-4f6f-bf55-e0f3d889691b 2024-09-11T05:52:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-11T05:52:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.export 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.export 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.export 2024-09-11T05:52:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.export 2024-09-11T05:52:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.export 2024-09-11T05:52:08.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.export 2024-09-11T05:52:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_38.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_38.export 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-11T05:52:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-11T05:52:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_39.status 2024-09-11T05:52:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.status 2024-09-11T05:52:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-11T05:52:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_39.status 2024-09-11T05:52:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-11T05:52:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-11T05:52:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 162 2024-09-11T05:52:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.status 2024-09-11T05:52:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.status 2024-09-11T05:52:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-11T05:52:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=162 2024-09-11T05:52:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T05:52:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=73 2024-09-11T05:52:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 162 = 73 2024-09-11T05:52:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T05:52:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 162 = '' 2024-09-11T05:52:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T05:52:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 162 = '' 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.status 2024-09-11T05:52:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T05:52:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=162 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 162 = 162 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-11T05:52:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_39': 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120bbf1da2f1 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120bbf1da2f1 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:14 2024 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:14 2024 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:14 2024 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c7239aa6-f45d-4b14-a8bb-4c545912a4a8 2024-09-11T05:52:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.export 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.export 2024-09-11T05:52:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.export 2024-09-11T05:52:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.export 2024-09-11T05:52:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.export 2024-09-11T05:52:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.export 2024-09-11T05:52:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_39.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_39.export 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-11T05:52:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-11T05:52:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-11T05:52:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-11T05:52:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-11T05:52:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_40.status 2024-09-11T05:52:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.status 2024-09-11T05:52:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-11T05:52:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_40.status 2024-09-11T05:52:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-11T05:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-11T05:52:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 164 2024-09-11T05:52:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.status 2024-09-11T05:52:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.status 2024-09-11T05:52:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-11T05:52:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=164 2024-09-11T05:52:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T05:52:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=74 2024-09-11T05:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 164 = 74 2024-09-11T05:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T05:52:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 164 = '' 2024-09-11T05:52:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T05:52:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 164 = '' 2024-09-11T05:52:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.status 2024-09-11T05:52:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T05:52:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=164 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 164 = 164 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_40': 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b2ef1f190 2024-09-11T05:52:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b2ef1f190 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:23 2024 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:23 2024 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:23 2024 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d6b90507-14e6-446f-bf0f-ecb9a6a2cc29 2024-09-11T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.export 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.export 2024-09-11T05:52:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.export 2024-09-11T05:52:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.export 2024-09-11T05:52:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.export 2024-09-11T05:52:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.export 2024-09-11T05:52:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_40.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_40.export 2024-09-11T05:52:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-11T05:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-11T05:52:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-11T05:52:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-11T05:52:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_41.status 2024-09-11T05:52:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.status 2024-09-11T05:52:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-11T05:52:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_41.status 2024-09-11T05:52:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-11T05:52:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 166 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.status 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.status 2024-09-11T05:52:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-11T05:52:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=166 2024-09-11T05:52:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T05:52:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=75 2024-09-11T05:52:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 166 = 75 2024-09-11T05:52:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T05:52:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 166 = '' 2024-09-11T05:52:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T05:52:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 166 = '' 2024-09-11T05:52:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.status 2024-09-11T05:52:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T05:52:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=166 2024-09-11T05:52:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 166 = 166 2024-09-11T05:52:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-11T05:52:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_41': 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120ba21a3651 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120ba21a3651 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:33 2024 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:33 2024 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:33 2024 2024-09-11T05:52:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f6dadc26-b728-4e0a-b9ab-f8acb9551249 2024-09-11T05:52:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.export 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.export 2024-09-11T05:52:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.export 2024-09-11T05:52:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.export 2024-09-11T05:52:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.export 2024-09-11T05:52:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.export 2024-09-11T05:52:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_41.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_41.export 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-11T05:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-11T05:52:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_42.status 2024-09-11T05:52:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.status 2024-09-11T05:52:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-11T05:52:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_42.status 2024-09-11T05:52:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-11T05:52:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-11T05:52:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 168 2024-09-11T05:52:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.status 2024-09-11T05:52:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.status 2024-09-11T05:52:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-11T05:52:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:38.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=168 2024-09-11T05:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T05:52:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=76 2024-09-11T05:52:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 168 = 76 2024-09-11T05:52:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T05:52:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 168 = '' 2024-09-11T05:52:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T05:52:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 168 = '' 2024-09-11T05:52:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.status 2024-09-11T05:52:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T05:52:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=168 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 168 = 168 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-11T05:52:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_42': 2024-09-11T05:52:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b8cde987c 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b8cde987c 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:46 2024 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:46 2024 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:46 2024 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 31052fdc-4c34-4a83-9382-8514a3004673 2024-09-11T05:52:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-11T05:52:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.export 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.export 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.export 2024-09-11T05:52:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.export 2024-09-11T05:52:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.export 2024-09-11T05:52:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.export 2024-09-11T05:52:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_42.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_42.export 2024-09-11T05:52:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-11T05:52:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:46.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7292338572257461) > weight (0.25) 2024-09-11T05:52:46.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.40464916131027107) > weight (0.25) 2024-09-11T05:52:46.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34084484171318974) > weight (0.25) 2024-09-11T05:52:46.588 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.858426852138841) > weight (0.25) 2024-09-11T05:52:46.588 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-11T05:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-11T05:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-11T05:52:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-11T05:52:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-11T05:52:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_43.status 2024-09-11T05:52:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.status 2024-09-11T05:52:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-11T05:52:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_43.status 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:47.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-11T05:52:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 170 2024-09-11T05:52:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.status 2024-09-11T05:52:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.status 2024-09-11T05:52:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-11T05:52:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=170 2024-09-11T05:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T05:52:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=77 2024-09-11T05:52:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 170 = 77 2024-09-11T05:52:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T05:52:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 170 = '' 2024-09-11T05:52:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T05:52:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 170 = '' 2024-09-11T05:52:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:49.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:52:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.status 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T05:52:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=170 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 170 = 170 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:52:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_43': 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120bde08dd77 2024-09-11T05:52:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120bde08dd77 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:44:53 2024 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:44:53 2024 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:44:53 2024 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d388f747-dd90-4450-9037-62799125d163 2024-09-11T05:52:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:52:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-11T05:52:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.export 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.export 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.export 2024-09-11T05:52:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.export 2024-09-11T05:52:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.export 2024-09-11T05:52:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:52:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.export 2024-09-11T05:52:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_43.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_43.export 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:52:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-11T05:52:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:52:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:52:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:52:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:52:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:52:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:52:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:52:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:52:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:52:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:52:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:52:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:52:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-11T05:52:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:52:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:52:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:52:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:52:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:52:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:52:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_44.status 2024-09-11T05:52:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.status 2024-09-11T05:52:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-11T05:52:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_44.status 2024-09-11T05:52:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-11T05:52:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 172 2024-09-11T05:52:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.status 2024-09-11T05:52:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:52:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.status 2024-09-11T05:52:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-11T05:52:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:52:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=172 2024-09-11T05:52:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:52:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T05:52:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=78 2024-09-11T05:52:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 172 = 78 2024-09-11T05:52:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:52:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T05:52:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 172 = '' 2024-09-11T05:52:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:52:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:52:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:52:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:52:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:52:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T05:52:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:52:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:52:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:52:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:52:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:52:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:52:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:52:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 172 = '' 2024-09-11T05:52:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:52:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:53:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:53:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.status 2024-09-11T05:53:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T05:53:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=172 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 172 = 172 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-11T05:53:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_44': 2024-09-11T05:53:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120bc03c230e 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120bc03c230e 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:03 2024 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:03 2024 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:03 2024 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0af1ac05-71b0-44c5-9afe-6c451a66416b 2024-09-11T05:53:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.export 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.export 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.export 2024-09-11T05:53:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.export 2024-09-11T05:53:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.export 2024-09-11T05:53:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.export 2024-09-11T05:53:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_44.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_44.export 2024-09-11T05:53:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2024-09-11T05:53:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-11T05:53:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-11T05:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-11T05:53:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-11T05:53:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-11T05:53:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-11T05:53:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_45.status 2024-09-11T05:53:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.status 2024-09-11T05:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-11T05:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_45.status 2024-09-11T05:53:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-11T05:53:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-11T05:53:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 174 2024-09-11T05:53:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.status 2024-09-11T05:53:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.status 2024-09-11T05:53:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-11T05:53:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=174 2024-09-11T05:53:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T05:53:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=79 2024-09-11T05:53:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 174 = 79 2024-09-11T05:53:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T05:53:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 174 = '' 2024-09-11T05:53:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T05:53:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 174 = '' 2024-09-11T05:53:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.status 2024-09-11T05:53:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T05:53:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=174 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 174 = 174 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_45': 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120bd7625643 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120bd7625643 2024-09-11T05:53:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:12 2024 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:12 2024 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:12 2024 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: eec72d56-6d39-496e-a649-5ff1b2b56355 2024-09-11T05:53:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.export 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.export 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.export 2024-09-11T05:53:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.export 2024-09-11T05:53:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.export 2024-09-11T05:53:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.export 2024-09-11T05:53:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_45.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_45.export 2024-09-11T05:53:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2024-09-11T05:53:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-11T05:53:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-11T05:53:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-11T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_46.status 2024-09-11T05:53:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.status 2024-09-11T05:53:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-11T05:53:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_46.status 2024-09-11T05:53:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-11T05:53:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-11T05:53:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 176 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.status 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.status 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-11T05:53:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=176 2024-09-11T05:53:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T05:53:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=80 2024-09-11T05:53:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 176 = 80 2024-09-11T05:53:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T05:53:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 176 = '' 2024-09-11T05:53:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T05:53:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 176 = '' 2024-09-11T05:53:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.status 2024-09-11T05:53:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T05:53:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=176 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 176 = 176 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_46': 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b593ced8e 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b593ced8e 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:18 2024 2024-09-11T05:53:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:18 2024 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:18 2024 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 854d50c4-dd91-4b99-9272-800916647ca1 2024-09-11T05:53:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-11T05:53:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:53:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.export 2024-09-11T05:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.export 2024-09-11T05:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.export 2024-09-11T05:53:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.export 2024-09-11T05:53:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.export 2024-09-11T05:53:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.export 2024-09-11T05:53:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_46.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_46.export 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-11T05:53:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-11T05:53:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-11T05:53:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-11T05:53:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_47.status 2024-09-11T05:53:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.status 2024-09-11T05:53:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-11T05:53:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_47.status 2024-09-11T05:53:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-11T05:53:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 178 2024-09-11T05:53:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.status 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.status 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-11T05:53:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=178 2024-09-11T05:53:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T05:53:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=81 2024-09-11T05:53:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 178 = 81 2024-09-11T05:53:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T05:53:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 178 = '' 2024-09-11T05:53:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T05:53:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 178 = '' 2024-09-11T05:53:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.status 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T05:53:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=178 2024-09-11T05:53:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 178 = 178 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_47': 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b29bda7b7 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b29bda7b7 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:26 2024 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:26 2024 2024-09-11T05:53:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:26 2024 2024-09-11T05:53:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9e3a1edf-65b1-4acf-aa6f-d0e474a04d1d 2024-09-11T05:53:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-11T05:53:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.export 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.export 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.export 2024-09-11T05:53:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.export 2024-09-11T05:53:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.export 2024-09-11T05:53:29.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.export 2024-09-11T05:53:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_47.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_47.export 2024-09-11T05:53:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-11T05:53:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-11T05:53:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-11T05:53:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-11T05:53:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_48.status 2024-09-11T05:53:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.status 2024-09-11T05:53:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-11T05:53:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_48.status 2024-09-11T05:53:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-11T05:53:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 180 2024-09-11T05:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.status 2024-09-11T05:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.status 2024-09-11T05:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-11T05:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=180 2024-09-11T05:53:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T05:53:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=82 2024-09-11T05:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 180 = 82 2024-09-11T05:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T05:53:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 180 = '' 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.status 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T05:53:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=180 2024-09-11T05:53:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 180 = 180 2024-09-11T05:53:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-11T05:53:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_48': 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b96c567e 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b96c567e 2024-09-11T05:53:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:32 2024 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:32 2024 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:32 2024 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5a84d065-9a68-4a66-a745-37e33ab34e73 2024-09-11T05:53:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.export 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.export 2024-09-11T05:53:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.export 2024-09-11T05:53:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.export 2024-09-11T05:53:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.export 2024-09-11T05:53:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.export 2024-09-11T05:53:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_48.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_48.export 2024-09-11T05:53:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-11T05:53:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-11T05:53:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-11T05:53:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-11T05:53:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_49.status 2024-09-11T05:53:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.status 2024-09-11T05:53:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-11T05:53:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_49.status 2024-09-11T05:53:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-11T05:53:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 182 2024-09-11T05:53:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.status 2024-09-11T05:53:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.status 2024-09-11T05:53:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-11T05:53:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=182 2024-09-11T05:53:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T05:53:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=83 2024-09-11T05:53:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 182 = 83 2024-09-11T05:53:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T05:53:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 182 = '' 2024-09-11T05:53:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T05:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 182 = '' 2024-09-11T05:53:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.status 2024-09-11T05:53:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T05:53:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=182 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 182 = 182 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:43.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9128730871637657) > weight (0.25) 2024-09-11T05:53:43.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41665573344931817) > weight (0.25) 2024-09-11T05:53:43.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8262228820564543) > weight (0.25) 2024-09-11T05:53:43.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.61215986364553) > weight (0.25) 2024-09-11T05:53:43.711 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-09-11T05:53:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-11T05:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_49': 2024-09-11T05:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120b64162540 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120b64162540 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:38 2024 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:38 2024 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:38 2024 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c3b8fd00-067d-4de2-881a-8a00643e7a65 2024-09-11T05:53:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.export 2024-09-11T05:53:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.export 2024-09-11T05:53:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.export 2024-09-11T05:53:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.export 2024-09-11T05:53:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.export 2024-09-11T05:53:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.export 2024-09-11T05:53:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_49.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_49.export 2024-09-11T05:53:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2024-09-11T05:53:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-11T05:53:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T05:53:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2024-09-11T05:53:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2024-09-11T05:53:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_instance=cluster1 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -i state 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance=cluster1 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local loop_instance 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T05:53:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sleep 0 2024-09-11T05:53:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T05:53:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ seq 0 3 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1:0 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ shift 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local val=cluster1:0 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster_var_name=cluster 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local instance_var_name=instance 2024-09-11T05:53:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval cluster=cluster1 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ eval instance=0 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local local_cluster=cluster1:0 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster=cluster1 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local val=cluster1:0 2024-09-11T05:53:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ local instance_var_name=instance 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ cluster=cluster1 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ instance=0 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval local_cluster=cluster1 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ local_cluster=cluster1 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ eval instance=0 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ instance=0 2024-09-11T05:53:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T05:53:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T05:53:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-11T05:53:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ return 0 2024-09-11T05:53:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T05:53:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo ' "state": "Replaying"' 2024-09-11T05:53:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2024-09-11T05:53:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2024-09-11T05:53:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2024-09-11T05:53:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2024-09-11T05:53:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = journal ']' 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-11T05:53:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster2-mirror-image_50.status 2024-09-11T05:53:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.status 2024-09-11T05:53:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-11T05:53:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ echo cluster1-mirror-image_50.status 2024-09-11T05:53:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T05:53:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-11T05:53:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-11T05:53:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 184 2024-09-11T05:53:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.status 2024-09-11T05:53:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.status 2024-09-11T05:53:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-11T05:53:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T05:53:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=184 2024-09-11T05:53:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2024-09-11T05:53:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T05:53:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=84 2024-09-11T05:53:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 184 = 84 2024-09-11T05:53:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2024-09-11T05:53:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T05:53:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 184 = '' 2024-09-11T05:53:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2024-09-11T05:53:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T05:53:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:53:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2024-09-11T05:53:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2024-09-11T05:53:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2024-09-11T05:53:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2024-09-11T05:53:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 184 = '' 2024-09-11T05:53:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T05:53:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.status 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T05:53:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T05:53:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T05:53:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=184 2024-09-11T05:53:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 184 = 184 2024-09-11T05:53:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T05:53:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T05:53:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-11T05:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_50': 2024-09-11T05:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2024-09-11T05:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2024-09-11T05:53:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120bfc8ae5b1 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120bfc8ae5b1 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:45:45 2024 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:45:45 2024 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:45:45 2024 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2f74885b-b07d-4f8f-aa98-e0b0749a2019 2024-09-11T05:53:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2024-09-11T05:53:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-11T05:53:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.export 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.export 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.export 2024-09-11T05:53:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.export 2024-09-11T05:53:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.export 2024-09-11T05:53:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T05:53:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.export 2024-09-11T05:53:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.Ange9SOiZn/cluster2-mirror-image_50.export /tmp/tmp.Ange9SOiZn/cluster1-mirror-image_50.export 2024-09-11T05:53:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: delete many images' 2024-09-11T05:53:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:53:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:53:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:53:54 TEST: delete many images 2024-09-11T05:53:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:53:54 TEST: delete many images 2024-09-11T05:53:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 50 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:53:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-11T05:53:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:53:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T05:53:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:53:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T05:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:53:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T05:53:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T05:53:55.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:53:55.203+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:53:55.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:53:55.203+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b4608500 [3/0081d803-369d-4214-80b0-726959bc08b4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:53:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:53:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:53:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:53:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2024-09-11T05:53:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-11T05:53:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:53:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:53:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-11T05:53:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:53:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:53:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T05:53:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:53:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:53:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:53:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T05:53:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T05:53:58.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:53:58.231+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:53:58.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:53:58.231+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ec780 [3/8de57868-2d26-4418-9611-9fe7ca6aec5d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:54:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-11T05:54:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T05:54:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:54:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T05:54:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:54:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T05:54:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T05:54:01.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:01.239+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:01.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:01.239+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47eca00 [3/938fc742-34c2-4279-9e72-cc118f790d8f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-11T05:54:03.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T05:54:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:54:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T05:54:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T05:54:04.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:04.491+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b4595520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:04.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:04.491+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b4608a00 [3/222ccbb3-07f2-4daf-aca6-881dd616be7f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-11T05:54:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T05:54:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:54:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T05:54:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:54:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T05:54:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T05:54:07.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:07.507+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:07.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:07.507+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b4609400 [3/4109d16e-0d23-49cb-8378-af90cd53b465] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:08.707 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb069520 2024-09-11T05:54:08.707 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb36cea0 2024-09-11T05:54:08.707 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556eebd05790 2024-09-11T05:54:08.707 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556eedde5930 2024-09-11T05:54:08.707 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556eee5b04e0 2024-09-11T05:54:08.708 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:08.704+0000 7f67c4fd4640 -1 reset not still connected to 0x556ef05b5930 2024-09-11T05:54:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2024-09-11T05:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-11T05:54:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:54:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-11T05:54:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T05:54:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:54:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T05:54:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T05:54:10.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:10.275+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b45911e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:10.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:10.275+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b48d2500 [3/fafb8d11-da7d-4858-9912-7044d668ce0c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:10.783+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b45791e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:10.783+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b48d2500 [3/fafb8d11-da7d-4858-9912-7044d668ce0c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-11T05:54:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T05:54:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:54:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T05:54:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:54:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T05:54:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T05:54:13.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:13.379+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b4591d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:13.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:13.379+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b48d2280 [3/f6c4c95b-26d9-427d-9fd8-9b7fe2a1b185] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:14.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.45222715327873175) > weight (0.25) 2024-09-11T05:54:14.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4798238127828861) > weight (0.25) 2024-09-11T05:54:14.589 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9008006344619945) > weight (0.25) 2024-09-11T05:54:14.589 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-11T05:54:14.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-11T05:54:14.590 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:54:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T05:54:14.587+0000 7f53fdfdd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 46956) UID: 1000 2024-09-11T05:54:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:54:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-11T05:54:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:16.108 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7a21c8a90 2024-09-11T05:54:16.109 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7a21ca270 2024-09-11T05:54:16.109 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7a2543930 2024-09-11T05:54:16.109 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7a2d8aa90 2024-09-11T05:54:16.109 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7abff69c0 2024-09-11T05:54:16.110 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b1adcdd0 2024-09-11T05:54:16.110 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b1ae24e0 2024-09-11T05:54:16.110 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b4a0ef70 2024-09-11T05:54:16.110 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.107+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b7b58680 2024-09-11T05:54:16.115 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.115+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7b8bb1d40 2024-09-11T05:54:16.116 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.115+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7bcf65a00 2024-09-11T05:54:16.117 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.115+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7c108cf70 2024-09-11T05:54:16.117 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.115+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7c1f2f380 2024-09-11T05:54:16.117 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2024-09-11T05:54:16.115+0000 7f41fb6d6640 -1 reset not still connected to 0x55e7c2802270 2024-09-11T05:54:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T05:54:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T05:54:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:54:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T05:54:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T05:54:16.267 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.264+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeac43040 2024-09-11T05:54:16.267 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.264+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb068c30 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb069380 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb069450 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb36cea0 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556eebd05790 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556eedde5930 2024-09-11T05:54:16.269 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:16.268+0000 7f67c4fd4640 -1 reset not still connected to 0x556ef05b5930 2024-09-11T05:54:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:16.343+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b457f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:16.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:16.343+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ecf00 [3/ab674d81-64fd-4a72-a384-57c22ab78502] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2024-09-11T05:54:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-11T05:54:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:54:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-11T05:54:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T05:54:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T05:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T05:54:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T05:54:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:19.342+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:19.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:19.342+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b4608c80 [3/29554ac5-9f7f-47c5-8ec4-11f432ce8b61] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:20.591 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-11T05:54:20.591 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:54:20.591 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-11T05:54:20.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T05:54:20.592 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T05:54:20.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T05:54:20.595 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-11T05:54:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-11T05:54:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T05:54:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:54:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T05:54:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T05:54:22.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:22.634+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:22.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:22.634+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b4609680 [3/417b48dd-e251-40f1-872a-99ad8bf806db] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:54:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-11T05:54:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_11 2024-09-11T05:54:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:54:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-11T05:54:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-11T05:54:25.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:25.370+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:25.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:25.370+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b48d2000 [3/f5febb24-4281-4e75-9b77-08bba442bf19] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:54:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-11T05:54:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_12 2024-09-11T05:54:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:54:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-11T05:54:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-11T05:54:28.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:28.398+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b4595380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:28.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:28.398+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ed180 [3/b16094f1-b9fc-4dcb-bceb-0331cdd63e45] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-11T05:54:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_13 2024-09-11T05:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-11T05:54:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-11T05:54:31.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:31.426+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b45801a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:31.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:31.426+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ecc80 [3/a98d5b12-6ee4-49e3-b583-ddcaf0392d73] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:54:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-11T05:54:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-11T05:54:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_14 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:54:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-11T05:54:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-11T05:54:34.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:34.162+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b457f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:34.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:34.162+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ec000 [3/5e3847d7-8152-4a70-94e2-5fa8963bddcf] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:35.797 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb068c30 2024-09-11T05:54:35.797 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb069380 2024-09-11T05:54:35.797 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb069450 2024-09-11T05:54:35.797 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eeb36cea0 2024-09-11T05:54:35.798 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eebd05790 2024-09-11T05:54:35.798 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556eedde5930 2024-09-11T05:54:35.798 INFO:tasks.ceph.cluster1.osd.0.smithi067.stderr:2024-09-11T05:54:35.795+0000 7f67c4fd4640 -1 reset not still connected to 0x556ef05b5930 2024-09-11T05:54:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-11T05:54:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-11T05:54:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:54:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_15 2024-09-11T05:54:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:54:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-11T05:54:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-11T05:54:37.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:37.178+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b45884e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:37.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:37.178+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ec280 [3/61ff442c-b39c-42e8-bcaf-1b8c56417011] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2024-09-11T05:54:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-11T05:54:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:54:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-11T05:54:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_16 2024-09-11T05:54:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:54:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-11T05:54:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-11T05:54:40.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:40.198+0000 7f4ad8c8b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b457eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:40.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:40.198+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b47edb80 [3/e19e65a3-67e6-4b67-bbe6-97b140e081a0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:40.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:40.762+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b4591380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:40.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:40.766+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b47edb80 [3/e19e65a3-67e6-4b67-bbe6-97b140e081a0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-11T05:54:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-11T05:54:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:54:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_17 2024-09-11T05:54:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-11T05:54:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-11T05:54:43.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:43.158+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b4590ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:43.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:43.158+0000 7f4adfc99640 -1 rbd::mirror::ImageReplayer: 0x5644b4609b80 [3/4bd5775b-ee7a-4c2e-8f7e-00eef1a21b9e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:54:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-11T05:54:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_18 2024-09-11T05:54:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:54:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-11T05:54:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-11T05:54:46.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:46.238+0000 7f4ad948c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5644b458eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:46.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:46.238+0000 7f4adf498640 -1 rbd::mirror::ImageReplayer: 0x5644b47ec500 [3/7f727691-cb67-470b-9ad0-990f37a8f164] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:47.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5950149823937672) > weight (0.25) 2024-09-11T05:54:47.712 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9693195733866569) > weight (0.25) 2024-09-11T05:54:47.712 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7399148322507365) > weight (0.25) 2024-09-11T05:54:47.712 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T05:54:47.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T05:54:47.713 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T05:54:47.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:54:47.714+0000 7f4ae049a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 37371) UID: 1000 2024-09-11T05:54:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-11T05:54:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_19 2024-09-11T05:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-11T05:54:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-11T05:54:49.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:49.266+0000 7f25cae18640 -1 rbd::mirror::ImageMap: 0x5574119e8c60 handle_peer_ack_remove: failed to remove global_image_id=18a350ae-4c11-4211-aa64-b6afa6f108ba 2024-09-11T05:54:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-11T05:54:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-11T05:54:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:54:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_20 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:54:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-11T05:54:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-11T05:54:52.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:52.262+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:52.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:52.262+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f32280 [3/4607f120-f3a3-4ead-9c81-2256f6efd774] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:53.714 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T05:54:53.714 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:54:53.715 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T05:54:53.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:54:53.715 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:54:53.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:54:53.718 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-11T05:54:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2024-09-11T05:54:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-11T05:54:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:54:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-11T05:54:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-11T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_21 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:54:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-11T05:54:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-11T05:54:55.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:55.302+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c48ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:55.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:55.302+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f97400 [3/d6371132-753a-4a4f-8524-8425689888b8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:54:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2024-09-11T05:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-11T05:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:54:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:54:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-11T05:54:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:54:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:54:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_22 2024-09-11T05:54:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:54:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:54:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:54:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-11T05:54:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-11T05:54:58.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:58.178+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c491e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:54:58.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:54:58.178+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f96500 [3/db729295-01a4-4d5f-aec1-17d8b32aefee] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:55:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-11T05:55:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_23 2024-09-11T05:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:55:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-11T05:55:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-11T05:55:01.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:01.362+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:01.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:01.362+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f96a00 [3/b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:01.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:01.898+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c49d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:01.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:01.898+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f96a00 [3/b43b2bd0-0e49-43aa-bb45-29d1ad7e93b7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:55:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-11T05:55:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-11T05:55:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:55:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_24 2024-09-11T05:55:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:55:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-11T05:55:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-11T05:55:04.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:04.365+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:04.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:04.365+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f32f00 [3/73f51a71-2b0c-4c23-b0d3-ddf1f35a1a67] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-11T05:55:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-11T05:55:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:55:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_25 2024-09-11T05:55:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:55:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-11T05:55:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-11T05:55:07.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:07.389+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3a820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:07.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:07.389+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f96280 [3/a87873a5-1cad-45f5-a638-56e77deab3e3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-11T05:55:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-11T05:55:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:55:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_26 2024-09-11T05:55:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:55:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-11T05:55:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-11T05:55:10.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:10.173+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:10.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:10.173+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f33900 [3/95e24eb9-a635-4d8b-b286-5bc01f045ba1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:55:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-11T05:55:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_27 2024-09-11T05:55:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:55:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-11T05:55:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-11T05:55:13.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:13.169+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:13.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:13.169+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411ec4c80 [3/1641c3d7-e0a2-407c-a874-91a1ef797a0c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-11T05:55:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_28 2024-09-11T05:55:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:55:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-11T05:55:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-11T05:55:16.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:16.225+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55741219f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:16.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:16.225+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411ec5400 [3/240aff3e-f4c4-4e78-89e9-f002bb8ec31b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:55:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-11T05:55:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_29 2024-09-11T05:55:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:55:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-11T05:55:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-11T05:55:19.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:19.241+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:19.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:19.241+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411ec5680 [3/2c6de312-17ab-498c-82cf-a905326cbb84] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-11T05:55:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_30 2024-09-11T05:55:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:55:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-11T05:55:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-11T05:55:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:22.253+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55741219e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:22.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:22.253+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411ec5180 [3/1c1a7c61-6e2b-48cc-80d9-2dcd7ffe860a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-11T05:55:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_31 2024-09-11T05:55:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:55:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-11T05:55:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-11T05:55:25.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:25.273+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:25.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:25.273+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f32780 [3/57412544-618d-4457-b99f-bd8f708b1bd6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-11T05:55:27.596 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9470276646886656) > weight (0.25) 2024-09-11T05:55:27.596 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4203012593565574) > weight (0.25) 2024-09-11T05:55:27.596 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7912651580035107) > weight (0.25) 2024-09-11T05:55:27.596 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.586957716781182) > weight (0.25) 2024-09-11T05:55:27.596 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-11T05:55:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_32 2024-09-11T05:55:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:55:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-11T05:55:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-11T05:55:28.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:28.173+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f3aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:28.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:28.173+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f96000 [3/a4730635-d6c8-461d-89b9-ef37acfa7eec] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:30.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-11T05:55:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-11T05:55:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:55:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_33 2024-09-11T05:55:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:55:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-11T05:55:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-11T05:55:31.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:31.325+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:31.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:31.325+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411370280 [3/fe60b457-0a30-4f36-a30e-adc07ffaf32e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:31.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:31.893+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f856c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:31.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:31.893+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411370280 [3/fe60b457-0a30-4f36-a30e-adc07ffaf32e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-11T05:55:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-11T05:55:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:55:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_34 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:55:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-11T05:55:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-11T05:55:34.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:34.337+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:34.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:34.337+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f33180 [3/82196961-27ae-4fd0-a78e-c8801a4e74b1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-11T05:55:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_35 2024-09-11T05:55:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:55:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-11T05:55:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-11T05:55:37.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:37.373+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c49380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:37.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:37.373+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f32500 [3/4c30e781-f967-4701-9b49-4258268b118a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-11T05:55:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_36 2024-09-11T05:55:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:55:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-11T05:55:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-11T05:55:40.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:40.181+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f85d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:40.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:40.181+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f33400 [3/9c4bb36c-7150-40d3-a228-8a4295768e81] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-11T05:55:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_37 2024-09-11T05:55:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:55:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-11T05:55:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-11T05:55:43.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:43.189+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:43.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:43.189+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f97b80 [3/f496dce6-4d07-4bf7-9ba7-d6c8866c01dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2024-09-11T05:55:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-11T05:55:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:55:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-11T05:55:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_38 2024-09-11T05:55:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:55:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-11T05:55:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-11T05:55:46.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:46.225+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:46.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:46.225+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f32c80 [3/62c8df86-31da-4f6f-bf55-e0f3d889691b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-11T05:55:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-11T05:55:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:55:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_39 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:55:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-11T05:55:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-11T05:55:49.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:49.273+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f841a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:49.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:49.273+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f96f00 [3/c7239aa6-f45d-4b14-a8bb-4c545912a4a8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-11T05:55:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-11T05:55:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:55:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_40 2024-09-11T05:55:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:55:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-11T05:55:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-11T05:55:52.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:52.297+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:52.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:52.297+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f97680 [3/d6b90507-14e6-446f-bf0f-ecb9a6a2cc29] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-11T05:55:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_41 2024-09-11T05:55:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:55:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-11T05:55:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-11T05:55:55.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:55.284+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:55.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:55.284+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411370000 [3/f6dadc26-b728-4e0a-b9ab-f8acb9551249] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:55:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:55:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:55:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-11T05:55:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:55:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-11T05:55:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:55:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:55:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:55:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_42 2024-09-11T05:55:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:55:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:55:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:55:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-11T05:55:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-11T05:55:58.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:58.188+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:55:58.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:55:58.192+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f32000 [3/31052fdc-4c34-4a83-9382-8514a3004673] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2024-09-11T05:56:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-11T05:56:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:56:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-11T05:56:01.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_43 2024-09-11T05:56:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:56:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-11T05:56:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-11T05:56:01.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:01.316+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574119f5860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:01.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:01.316+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f97180 [3/d388f747-dd90-4450-9037-62799125d163] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:01.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:01.872+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f844e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:01.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:01.872+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f97180 [3/d388f747-dd90-4450-9037-62799125d163] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-11T05:56:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_44 2024-09-11T05:56:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:56:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-11T05:56:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-11T05:56:04.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:04.336+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574119f56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:04.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:04.336+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x5574106ba000 [3/0af1ac05-71b0-44c5-9afe-6c451a66416b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-11T05:56:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_45 2024-09-11T05:56:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:56:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-11T05:56:07.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-11T05:56:07.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:07.372+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574119f4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:07.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:07.372+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f97900 [3/eec72d56-6d39-496e-a649-5ff1b2b56355] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-11T05:56:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_46 2024-09-11T05:56:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:56:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-11T05:56:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-11T05:56:10.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:10.208+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c491e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:10.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:10.208+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f33680 [3/854d50c4-dd91-4b99-9272-800916647ca1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-11T05:56:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_47 2024-09-11T05:56:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:56:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-11T05:56:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-11T05:56:13.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:13.212+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c496c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:13.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:13.212+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f33b80 [3/9e3a1edf-65b1-4acf-aa6f-d0e474a04d1d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:56:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-11T05:56:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_48 2024-09-11T05:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:56:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-11T05:56:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-11T05:56:16.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:16.220+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574119f49c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:16.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:16.220+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f32a00 [3/5a84d065-9a68-4a66-a745-37e33ab34e73] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-11T05:56:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-11T05:56:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:56:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_49 2024-09-11T05:56:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:56:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-11T05:56:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-11T05:56:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:19.284+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574119f4820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:19.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:19.284+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f96c80 [3/c3b8fd00-067d-4de2-881a-8a00643e7a65] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:56:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2024-09-11T05:56:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-11T05:56:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:56:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_50 2024-09-11T05:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:56:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-11T05:56:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-11T05:56:22.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:22.280+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c49ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:56:22.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:56:22.280+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411ec5900 [3/2f74885b-b07d-4f8f-aa98-e0b0749a2019] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:56:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T05:56:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-11T05:56:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:56:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:56:24 TEST: image deletions should propagate 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:56:24 TEST: image deletions should propagate 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=0 2024-09-11T05:56:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2024-09-11T05:56:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2024-09-11T05:56:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2024-09-11T05:56:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:56:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:56:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:56:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T05:56:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'images: ' 2024-09-11T05:56:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 2 2024-09-11T05:56:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=0 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2024-09-11T05:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 40 2024-09-11T05:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in `seq 1 40` 2024-09-11T05:56:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2024-09-11T05:56:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T05:56:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep 'image health:' 2024-09-11T05:56:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cut '-d ' -f 3 2024-09-11T05:56:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T05:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2024-09-11T05:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2024-09-11T05:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2024-09-11T05:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 50 2024-09-11T05:56:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2024-09-11T05:56:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T05:56:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:56:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T05:56:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-11T05:56:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:56:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:56:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T05:56:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:56:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:56:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:56:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T05:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:56:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:56:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T05:56:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T05:56:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T05:56:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T05:56:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T05:56:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T05:56:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T05:56:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T05:56:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T05:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T05:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T05:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T05:56:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T05:56:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-11T05:56:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T05:56:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2024-09-11T05:56:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T05:56:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T05:56:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T05:56:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T05:56:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T05:56:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T05:56:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T05:56:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T05:56:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T05:56:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T05:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T05:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T05:56:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T05:56:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2024-09-11T05:56:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T05:56:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T05:56:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T05:56:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T05:56:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T05:56:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2024-09-11T05:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_11 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T05:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T05:56:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_11 2024-09-11T05:56:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T05:56:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T05:56:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_12 2024-09-11T05:56:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T05:56:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T05:56:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_12 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T05:56:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T05:56:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T05:56:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_13 2024-09-11T05:56:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T05:56:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T05:56:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-11T05:56:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2024-09-11T05:56:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_13 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T05:56:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T05:56:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_14 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T05:56:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T05:56:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-11T05:56:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_14 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T05:56:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T05:56:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_15 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T05:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T05:56:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-11T05:56:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_15 2024-09-11T05:56:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T05:56:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:56:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_16 2024-09-11T05:56:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T05:56:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T05:56:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T05:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_16 2024-09-11T05:56:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T05:56:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T05:56:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_17 2024-09-11T05:56:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T05:56:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T05:56:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_17 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T05:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T05:56:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T05:56:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_18 2024-09-11T05:56:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T05:56:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T05:56:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-11T05:56:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2024-09-11T05:56:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_18 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T05:56:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_19 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T05:56:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T05:56:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-11T05:56:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2024-09-11T05:56:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T05:56:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_19 2024-09-11T05:56:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T05:56:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_20 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T05:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T05:56:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T05:56:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_20 2024-09-11T05:56:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T05:56:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_21 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T05:56:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T05:56:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_21 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T05:56:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T05:56:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_22 2024-09-11T05:56:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T05:56:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T05:56:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T05:56:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_22 2024-09-11T05:56:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T05:56:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T05:56:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T05:56:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_23 2024-09-11T05:56:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T05:56:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T05:56:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T05:56:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_23 2024-09-11T05:56:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T05:56:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T05:56:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_24 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T05:56:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T05:56:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_24 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T05:56:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:56:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_25 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T05:56:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T05:56:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T05:56:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_25 2024-09-11T05:56:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T05:56:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T05:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_26 2024-09-11T05:56:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T05:56:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T05:56:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T05:56:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_26 2024-09-11T05:56:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T05:56:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T05:56:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_27 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T05:56:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T05:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-11T05:56:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T05:56:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_27 2024-09-11T05:56:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T05:56:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:56:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_28 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T05:56:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T05:56:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-11T05:56:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2024-09-11T05:56:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_28 2024-09-11T05:56:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T05:56:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T05:56:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_29 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T05:56:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T05:56:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T05:56:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_29 2024-09-11T05:56:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T05:56:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:56:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_30 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T05:56:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T05:56:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_30 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T05:56:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T05:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_31 2024-09-11T05:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T05:56:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T05:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-11T05:56:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2024-09-11T05:56:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_31 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T05:56:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_32 2024-09-11T05:56:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T05:56:30.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T05:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-11T05:56:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2024-09-11T05:56:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_32 2024-09-11T05:56:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T05:56:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T05:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_33 2024-09-11T05:56:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T05:56:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T05:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T05:56:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_33 2024-09-11T05:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T05:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T05:56:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_34 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T05:56:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T05:56:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-11T05:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2024-09-11T05:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_34 2024-09-11T05:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T05:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-11T05:56:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_35 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T05:56:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T05:56:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-11T05:56:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_35 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T05:56:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_36 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T05:56:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T05:56:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T05:56:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_36 2024-09-11T05:56:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T05:56:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2024-09-11T05:56:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_37 2024-09-11T05:56:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T05:56:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T05:56:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-11T05:56:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_37 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T05:56:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_38 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T05:56:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T05:56:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:31.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T05:56:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_38 2024-09-11T05:56:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T05:56:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-11T05:56:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_39 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T05:56:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T05:56:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_39 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T05:56:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-11T05:56:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_40 2024-09-11T05:56:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T05:56:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T05:56:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_40 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T05:56:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T05:56:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_41 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T05:56:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T05:56:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_41 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T05:56:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T05:56:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T05:56:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_42 2024-09-11T05:56:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T05:56:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T05:56:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_42 2024-09-11T05:56:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T05:56:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T05:56:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_43 2024-09-11T05:56:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T05:56:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T05:56:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-11T05:56:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_43 2024-09-11T05:56:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T05:56:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_44 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T05:56:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-11T05:56:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2024-09-11T05:56:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T05:56:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_44 2024-09-11T05:56:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T05:56:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T05:56:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T05:56:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_45 2024-09-11T05:56:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T05:56:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T05:56:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-11T05:56:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_45 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T05:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T05:56:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_46 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T05:56:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T05:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-11T05:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2024-09-11T05:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_46 2024-09-11T05:56:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T05:56:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T05:56:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T05:56:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_47 2024-09-11T05:56:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T05:56:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T05:56:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T05:56:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_47 2024-09-11T05:56:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T05:56:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T05:56:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T05:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_48 2024-09-11T05:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T05:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T05:56:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_48 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T05:56:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2024-09-11T05:56:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_49 2024-09-11T05:56:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T05:56:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T05:56:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_49 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T05:56:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T05:56:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_50 2024-09-11T05:56:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T05:56:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T05:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_50 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T05:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T05:56:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T05:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 05:56:34 TEST: delete images during bootstrap 2024-09-11T05:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 05:56:34 TEST: delete images during bootstrap 2024-09-11T05:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-11T05:56:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=image 2024-09-11T05:56:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-11T05:56:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd: mirroring is already configured for image mode 2024-09-11T05:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-11T05:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=image 2024-09-11T05:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-11T05:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd: mirroring is already configured for image mode 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=cluster1 2024-09-11T05:56:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ cluster=cluster1 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ instance=0 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=test 2024-09-11T05:56:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 1 10 2024-09-11T05:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2024-09-11T05:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-11T05:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:56:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:56:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-11T05:56:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:56:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1904c8e78dbc 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1904c8e78dbc 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:56:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:56:34 2024 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:56:34 2024 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:56:34 2024 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 084b7d80-d6c3-4223-9158-cf80b853bf1e 2024-09-11T05:56:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:56:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T05:56:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:56:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:56:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:56:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:56:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 38043648 bytes 2024-09-11T05:56:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 83078144 bytes 2024-09-11T05:56:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 115961856 bytes 2024-09-11T05:56:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 140481536 bytes 2024-09-11T05:56:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 247830528 bytes 2024-09-11T05:56:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 281694208 bytes 2024-09-11T05:56:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 329139200 bytes 2024-09-11T05:56:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 366649344 bytes 2024-09-11T05:56:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 391266304 bytes 2024-09-11T05:56:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 446362624 bytes 2024-09-11T05:56:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 485610496 bytes 2024-09-11T05:56:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 513283072 bytes 2024-09-11T05:56:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 542351360 bytes 2024-09-11T05:56:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 566845440 bytes 2024-09-11T05:56:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 635772928 bytes 2024-09-11T05:56:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 697493504 bytes 2024-09-11T05:56:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 714877952 bytes 2024-09-11T05:56:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 733673472 bytes 2024-09-11T05:56:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 759696384 bytes 2024-09-11T05:56:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 760327168 bytes 2024-09-11T05:56:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 785704960 bytes 2024-09-11T05:56:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 793640960 bytes 2024-09-11T05:56:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 830900224 bytes 2024-09-11T05:56:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 874923008 bytes 2024-09-11T05:56:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 881743872 bytes 2024-09-11T05:56:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 914895872 bytes 2024-09-11T05:56:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 926195712 bytes 2024-09-11T05:56:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 926812160 bytes 2024-09-11T05:56:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 972509184 bytes 2024-09-11T05:56:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1075096576 bytes 2024-09-11T05:56:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1162350592 bytes 2024-09-11T05:56:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1196750848 bytes 2024-09-11T05:56:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1290756096 bytes 2024-09-11T05:56:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1381853184 bytes 2024-09-11T05:56:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1388832768 bytes 2024-09-11T05:56:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1433185280 bytes 2024-09-11T05:56:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1509090304 bytes 2024-09-11T05:56:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1526642688 bytes 2024-09-11T05:56:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1631201280 bytes 2024-09-11T05:56:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1659144192 bytes 2024-09-11T05:56:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1682961408 bytes 2024-09-11T05:56:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1768587264 bytes 2024-09-11T05:56:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1830840320 bytes 2024-09-11T05:56:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1864630272 bytes 2024-09-11T05:56:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1924697088 bytes 2024-09-11T05:56:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1946967040 bytes 2024-09-11T05:56:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1993467904 bytes 2024-09-11T05:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2028138496 bytes 2024-09-11T05:56:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2029975552 bytes 2024-09-11T05:56:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2055352320 bytes 2024-09-11T05:56:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2153162752 bytes 2024-09-11T05:56:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2192831488 bytes 2024-09-11T05:56:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2221246464 bytes 2024-09-11T05:56:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2227010560 bytes 2024-09-11T05:56:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2270615552 bytes 2024-09-11T05:56:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2290237440 bytes 2024-09-11T05:56:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2313190400 bytes 2024-09-11T05:56:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2350007296 bytes 2024-09-11T05:56:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2398820352 bytes 2024-09-11T05:56:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2423629824 bytes 2024-09-11T05:56:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2441361408 bytes 2024-09-11T05:56:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2447134720 bytes 2024-09-11T05:56:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2478342144 bytes 2024-09-11T05:56:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2538317824 bytes 2024-09-11T05:56:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2569495552 bytes 2024-09-11T05:56:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2618999808 bytes 2024-09-11T05:56:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2638306304 bytes 2024-09-11T05:56:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2671762432 bytes 2024-09-11T05:56:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2706375680 bytes 2024-09-11T05:56:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2723571712 bytes 2024-09-11T05:56:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2732186624 bytes 2024-09-11T05:56:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2839655424 bytes 2024-09-11T05:56:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2872077312 bytes 2024-09-11T05:56:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2909869056 bytes 2024-09-11T05:56:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2921233408 bytes 2024-09-11T05:56:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2948775936 bytes 2024-09-11T05:56:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 2966694912 bytes 2024-09-11T05:56:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3001583616 bytes 2024-09-11T05:56:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3046227968 bytes 2024-09-11T05:56:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3080359936 bytes 2024-09-11T05:56:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3100822528 bytes 2024-09-11T05:56:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3158709248 bytes 2024-09-11T05:56:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3240118272 bytes 2024-09-11T05:56:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3285309440 bytes 2024-09-11T05:56:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3332001792 bytes 2024-09-11T05:56:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:56:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:56:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120b23411ea1 2024-09-11T05:56:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:56:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:56:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 present 120b23411ea1 2024-09-11T05:56:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:56:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T05:56:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120b23411ea1 2024-09-11T05:56:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:56:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:56:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120b23411ea1 2024-09-11T05:56:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120b23411ea1 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120b23411ea1 -o 120b23411ea1 = 120b23411ea1 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-11T05:56:51.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.589549104278584) > weight (0.25) 2024-09-11T05:56:51.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7073057543440664) > weight (0.25) 2024-09-11T05:56:51.719 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2706785363711228) > weight (0.25) 2024-09-11T05:56:51.719 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T05:56:51.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T05:56:51.720 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T05:56:51.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T05:56:51.719+0000 7fa781c99640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 55867) UID: 1000 2024-09-11T05:56:56.597 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7875313065253743) > weight (0.25) 2024-09-11T05:56:56.597 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-11T05:56:56.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-11T05:56:56.597 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:56:56.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T05:56:56.595+0000 7fb022777640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39066) UID: 1000 2024-09-11T05:56:58.720 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T05:56:58.721 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:56:58.721 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T05:56:58.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T05:56:58.722 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T05:56:58.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T05:56:58.725 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-11T05:56:59.600 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-11T05:56:59.600 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:56:59.600 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-11T05:56:59.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T05:56:59.601 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T05:56:59.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T05:56:59.604 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-09-11T05:57:01.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:57:01.863+0000 7f25c3e0a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411c49860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:57:01.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:57:01.863+0000 7f25cae18640 -1 rbd::mirror::ImageReplayer: 0x557411f33b80 [3/084b7d80-d6c3-4223-9158-cf80b853bf1e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:57:17.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:57:17.983+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ee2acc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-11T05:57:17.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:57:17.983+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ee2acc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=185, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-11T05:57:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:57:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:57:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:57:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T05:57:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:57:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:57:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:57:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:57:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T05:57:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:57:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:57:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_1 2024-09-11T05:57:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T05:57:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:57:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-11T05:57:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:57:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:57:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T05:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2024-09-11T05:57:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T05:57:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T05:57:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:57:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:57:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:57:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:57:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:57:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-11T05:57:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-11T05:57:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:57:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-11T05:57:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:57:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-11T05:57:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2024-09-11T05:57:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:57:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 192ba5d06759 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.192ba5d06759 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:57:18 2024 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:57:18 2024 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:57:18 2024 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:57:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:57:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b0f24933-38e1-4e99-be0b-91d431ee980b 2024-09-11T05:57:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:57:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T05:57:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:57:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2024-09-11T05:57:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:57:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:57:19.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 66407424 bytes 2024-09-11T05:57:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 120870912 bytes 2024-09-11T05:57:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 140901376 bytes 2024-09-11T05:57:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 213598208 bytes 2024-09-11T05:57:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 235359232 bytes 2024-09-11T05:57:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 274444288 bytes 2024-09-11T05:57:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 355297280 bytes 2024-09-11T05:57:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 416111616 bytes 2024-09-11T05:57:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 435784704 bytes 2024-09-11T05:57:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 479987712 bytes 2024-09-11T05:57:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 484448256 bytes 2024-09-11T05:57:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 526039040 bytes 2024-09-11T05:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 591082496 bytes 2024-09-11T05:57:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 630620160 bytes 2024-09-11T05:57:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 657413120 bytes 2024-09-11T05:57:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 797317120 bytes 2024-09-11T05:57:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 797918208 bytes 2024-09-11T05:57:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 825023488 bytes 2024-09-11T05:57:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 867635200 bytes 2024-09-11T05:57:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 926508032 bytes 2024-09-11T05:57:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 948218880 bytes 2024-09-11T05:57:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 993415168 bytes 2024-09-11T05:57:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1022309376 bytes 2024-09-11T05:57:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1047867392 bytes 2024-09-11T05:57:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1112295424 bytes 2024-09-11T05:57:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1182780416 bytes 2024-09-11T05:57:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1238582272 bytes 2024-09-11T05:57:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1249153024 bytes 2024-09-11T05:57:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1286582272 bytes 2024-09-11T05:57:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1321808896 bytes 2024-09-11T05:57:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1397978112 bytes 2024-09-11T05:57:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1425267712 bytes 2024-09-11T05:57:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1474608128 bytes 2024-09-11T05:57:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1485241344 bytes 2024-09-11T05:57:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1561860096 bytes 2024-09-11T05:57:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1656309760 bytes 2024-09-11T05:57:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1707078656 bytes 2024-09-11T05:57:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1710404608 bytes 2024-09-11T05:57:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1725973504 bytes 2024-09-11T05:57:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1830605824 bytes 2024-09-11T05:57:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1858783232 bytes 2024-09-11T05:57:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1907712000 bytes 2024-09-11T05:57:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1937658880 bytes 2024-09-11T05:57:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1989238784 bytes 2024-09-11T05:57:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2047252480 bytes 2024-09-11T05:57:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2135609344 bytes 2024-09-11T05:57:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2190087168 bytes 2024-09-11T05:57:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2225218560 bytes 2024-09-11T05:57:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2258277376 bytes 2024-09-11T05:57:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2338712576 bytes 2024-09-11T05:57:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2400681984 bytes 2024-09-11T05:57:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2408503296 bytes 2024-09-11T05:57:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2455333888 bytes 2024-09-11T05:57:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2483223552 bytes 2024-09-11T05:57:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2501449728 bytes 2024-09-11T05:57:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2531450880 bytes 2024-09-11T05:57:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2559986688 bytes 2024-09-11T05:57:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2583258112 bytes 2024-09-11T05:57:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2603720704 bytes 2024-09-11T05:57:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2685509632 bytes 2024-09-11T05:57:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2694389760 bytes 2024-09-11T05:57:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2695017472 bytes 2024-09-11T05:57:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2719678464 bytes 2024-09-11T05:57:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2742646784 bytes 2024-09-11T05:57:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2766770176 bytes 2024-09-11T05:57:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2784309248 bytes 2024-09-11T05:57:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2943828992 bytes 2024-09-11T05:57:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2952053760 bytes 2024-09-11T05:57:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2981337088 bytes 2024-09-11T05:57:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2989732864 bytes 2024-09-11T05:57:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3005553664 bytes 2024-09-11T05:57:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3086307328 bytes 2024-09-11T05:57:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3109690368 bytes 2024-09-11T05:57:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3115454464 bytes 2024-09-11T05:57:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3165600768 bytes 2024-09-11T05:57:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3191196672 bytes 2024-09-11T05:57:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3219549184 bytes 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:57:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120bd706e78a 2024-09-11T05:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:57:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 present 120bd706e78a 2024-09-11T05:57:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:57:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T05:57:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120bd706e78a 2024-09-11T05:57:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:57:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:57:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:57:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:57:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:57:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:57:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:57:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:57:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120bd706e78a 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120bd706e78a 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120bd706e78a -o 120bd706e78a = 120bd706e78a 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:57:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-11T05:57:31.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:57:31.862+0000 7f25c460b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557411f84820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:57:31.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:57:31.862+0000 7f25ca617640 -1 rbd::mirror::ImageReplayer: 0x557411f33680 [3/b0f24933-38e1-4e99-be0b-91d431ee980b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:58:01.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:58:01.078+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ee632c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-11T05:58:01.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:58:01.078+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ee632c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-11T05:58:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T05:58:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T05:58:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:58:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:58:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:58:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:58:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_2 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T05:58:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T05:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-11T05:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-11T05:58:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T05:58:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:58:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:58:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:58:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T05:58:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2024-09-11T05:58:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T05:58:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T05:58:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:58:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-11T05:58:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:58:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-11T05:58:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19445bae40df 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19445bae40df 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:58:02 2024 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:58:02 2024 2024-09-11T05:58:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:58:02 2024 2024-09-11T05:58:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:58:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:58:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c82773d9-6a6b-45aa-8ab1-a01d4219a50f 2024-09-11T05:58:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:58:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T05:58:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:58:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2024-09-11T05:58:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:58:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:58:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 642048 bytes 2024-09-11T05:58:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 44317696 bytes 2024-09-11T05:58:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 57900032 bytes 2024-09-11T05:58:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 76783616 bytes 2024-09-11T05:58:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 90621952 bytes 2024-09-11T05:58:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 138335232 bytes 2024-09-11T05:58:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 179294208 bytes 2024-09-11T05:58:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 191913984 bytes 2024-09-11T05:58:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 254039040 bytes 2024-09-11T05:58:04.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 262794240 bytes 2024-09-11T05:58:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 303287296 bytes 2024-09-11T05:58:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 361756672 bytes 2024-09-11T05:58:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 389735424 bytes 2024-09-11T05:58:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 415767552 bytes 2024-09-11T05:58:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 515155968 bytes 2024-09-11T05:58:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 569800704 bytes 2024-09-11T05:58:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 662750208 bytes 2024-09-11T05:58:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 696043520 bytes 2024-09-11T05:58:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 747406336 bytes 2024-09-11T05:58:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 785590272 bytes 2024-09-11T05:58:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 806639616 bytes 2024-09-11T05:58:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 846938112 bytes 2024-09-11T05:58:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 893503488 bytes 2024-09-11T05:58:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 925287424 bytes 2024-09-11T05:58:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1004495872 bytes 2024-09-11T05:58:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1079534592 bytes 2024-09-11T05:58:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1133015040 bytes 2024-09-11T05:58:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1195109376 bytes 2024-09-11T05:58:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1209451520 bytes 2024-09-11T05:58:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1258747904 bytes 2024-09-11T05:58:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1312782336 bytes 2024-09-11T05:58:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1340539904 bytes 2024-09-11T05:58:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1380847616 bytes 2024-09-11T05:58:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1442334720 bytes 2024-09-11T05:58:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1462377472 bytes 2024-09-11T05:58:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1462981632 bytes 2024-09-11T05:58:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1525217280 bytes 2024-09-11T05:58:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1559982080 bytes 2024-09-11T05:58:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1611130880 bytes 2024-09-11T05:58:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1676408832 bytes 2024-09-11T05:58:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1707469824 bytes 2024-09-11T05:58:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1742733312 bytes 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:58:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120b9fa8af79 2024-09-11T05:58:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:58:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:58:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 present 120b9fa8af79 2024-09-11T05:58:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T05:58:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120b9fa8af79 2024-09-11T05:58:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:58:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:58:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:58:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:58:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:58:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120b9fa8af79 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120b9fa8af79 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120b9fa8af79 -o 120b9fa8af79 = 120b9fa8af79 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-11T05:58:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-11T05:58:14.726 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T05:58:14.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T05:58:14.727 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T05:58:14.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T05:58:14.725+0000 7f25cb619640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48084) UID: 1000 2024-09-11T05:58:18.727 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T05:58:18.728 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:58:18.728 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T05:58:18.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T05:58:18.729 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T05:58:18.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T05:58:18.732 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-11T05:58:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:58:18.829+0000 7fe14d238640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f136a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:58:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:58:18.829+0000 7fe154246640 -1 rbd::mirror::ImageReplayer: 0x5624f1810780 [3/c82773d9-6a6b-45aa-8ab1-a01d4219a50f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:58:40.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:58:40.045+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ecac6400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-11T05:58:40.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:58:40.045+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ecac6400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-11T05:58:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T05:58:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T05:58:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:58:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:58:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T05:58:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:58:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:58:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:58:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_3 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T05:58:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T05:58:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:58:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:58:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:58:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T05:58:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2024-09-11T05:58:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T05:58:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T05:58:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T05:58:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:58:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:58:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:58:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:58:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-11T05:58:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-11T05:58:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:58:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-11T05:58:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:58:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 196626b9ef23 2024-09-11T05:58:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.196626b9ef23 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:58:40 2024 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:58:40 2024 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:58:40 2024 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1e8863ff-ce63-4fa8-b078-d0d8883ed61a 2024-09-11T05:58:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:58:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T05:58:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:58:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T05:58:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:58:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:58:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 59286528 bytes 2024-09-11T05:58:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 80288768 bytes 2024-09-11T05:58:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 154445824 bytes 2024-09-11T05:58:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 215279616 bytes 2024-09-11T05:58:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 276562944 bytes 2024-09-11T05:58:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 312462336 bytes 2024-09-11T05:58:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 328792064 bytes 2024-09-11T05:58:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 350587904 bytes 2024-09-11T05:58:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 465861632 bytes 2024-09-11T05:58:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 562319360 bytes 2024-09-11T05:58:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 619327488 bytes 2024-09-11T05:58:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 619950080 bytes 2024-09-11T05:58:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 633382912 bytes 2024-09-11T05:58:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 637042688 bytes 2024-09-11T05:58:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 662350848 bytes 2024-09-11T05:58:43.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4878169052587007) > weight (0.25) 2024-09-11T05:58:43.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.42759712312818465) > weight (0.25) 2024-09-11T05:58:43.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5743196910606749) > weight (0.25) 2024-09-11T05:58:43.604 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-11T05:58:43.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-11T05:58:43.605 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-11T05:58:43.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T05:58:43.605+0000 7fe9a9cf8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 55094) UID: 1000 2024-09-11T05:58:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 690498560 bytes 2024-09-11T05:58:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 714805248 bytes 2024-09-11T05:58:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 754663424 bytes 2024-09-11T05:58:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 797947904 bytes 2024-09-11T05:58:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 826835968 bytes 2024-09-11T05:58:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 886854656 bytes 2024-09-11T05:58:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 965912576 bytes 2024-09-11T05:58:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 982773760 bytes 2024-09-11T05:58:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1010207744 bytes 2024-09-11T05:58:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1026878464 bytes 2024-09-11T05:58:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1033044992 bytes 2024-09-11T05:58:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1053366272 bytes 2024-09-11T05:58:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1076486144 bytes 2024-09-11T05:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1130372096 bytes 2024-09-11T05:58:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1130973184 bytes 2024-09-11T05:58:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1155219456 bytes 2024-09-11T05:58:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1192503296 bytes 2024-09-11T05:58:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1244224512 bytes 2024-09-11T05:58:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1280133120 bytes 2024-09-11T05:58:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1360293888 bytes 2024-09-11T05:58:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:58:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:58:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:58:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120ba0112146 2024-09-11T05:58:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:58:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 present 120ba0112146 2024-09-11T05:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:58:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T05:58:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120ba0112146 2024-09-11T05:58:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:58:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:58:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120ba0112146 2024-09-11T05:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120ba0112146 2024-09-11T05:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120ba0112146 -o 120ba0112146 = 120ba0112146 2024-09-11T05:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T05:58:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:58:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-11T05:58:52.605 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-11T05:58:52.605 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:58:52.606 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-11T05:58:52.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T05:58:52.606 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T05:58:52.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T05:58:52.609 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-11T05:59:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:59:13.880+0000 7fe14da39640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f136ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:59:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:59:13.880+0000 7fe154246640 -1 rbd::mirror::ImageReplayer: 0x5624f1810a00 [3/1e8863ff-ce63-4fa8-b078-d0d8883ed61a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T05:59:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:59:18.064+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5f0cfcc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-11T05:59:18.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T05:59:18.064+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5f0cfcc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-11T05:59:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T05:59:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:59:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:59:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:59:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:59:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T05:59:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T05:59:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:59:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:59:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:59:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T05:59:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:59:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:59:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_4 2024-09-11T05:59:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T05:59:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:59:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:59:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T05:59:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T05:59:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T05:59:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2024-09-11T05:59:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T05:59:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T05:59:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:59:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T05:59:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-11T05:59:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-11T05:59:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T05:59:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-11T05:59:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T05:59:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1984fc6bd44f 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1984fc6bd44f 2024-09-11T05:59:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 05:59:18 2024 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 05:59:18 2024 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 05:59:18 2024 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f1f69ba8-2492-401d-abb0-04773b8fbe74 2024-09-11T05:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T05:59:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T05:59:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T05:59:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2024-09-11T05:59:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T05:59:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T05:59:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 48204800 bytes 2024-09-11T05:59:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 116615168 bytes 2024-09-11T05:59:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 147635200 bytes 2024-09-11T05:59:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 156986368 bytes 2024-09-11T05:59:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 179788800 bytes 2024-09-11T05:59:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 245203968 bytes 2024-09-11T05:59:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 274254848 bytes 2024-09-11T05:59:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 293302272 bytes 2024-09-11T05:59:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 321274880 bytes 2024-09-11T05:59:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 333597696 bytes 2024-09-11T05:59:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 380548096 bytes 2024-09-11T05:59:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 400552960 bytes 2024-09-11T05:59:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 408737792 bytes 2024-09-11T05:59:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 491762688 bytes 2024-09-11T05:59:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 536912896 bytes 2024-09-11T05:59:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 562920448 bytes 2024-09-11T05:59:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 568941568 bytes 2024-09-11T05:59:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 577285120 bytes 2024-09-11T05:59:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 649372672 bytes 2024-09-11T05:59:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 705475584 bytes 2024-09-11T05:59:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 757306368 bytes 2024-09-11T05:59:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 794572800 bytes 2024-09-11T05:59:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 816551936 bytes 2024-09-11T05:59:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 852602880 bytes 2024-09-11T05:59:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 954216448 bytes 2024-09-11T05:59:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 971738112 bytes 2024-09-11T05:59:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 977715200 bytes 2024-09-11T05:59:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 995542016 bytes 2024-09-11T05:59:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1014998016 bytes 2024-09-11T05:59:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1027913728 bytes 2024-09-11T05:59:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1074489344 bytes 2024-09-11T05:59:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1130859520 bytes 2024-09-11T05:59:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1202012160 bytes 2024-09-11T05:59:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1221470208 bytes 2024-09-11T05:59:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1235741696 bytes 2024-09-11T05:59:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1251026944 bytes 2024-09-11T05:59:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1303557120 bytes 2024-09-11T05:59:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1361972224 bytes 2024-09-11T05:59:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1408778240 bytes 2024-09-11T05:59:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1497496576 bytes 2024-09-11T05:59:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1527299072 bytes 2024-09-11T05:59:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1562606592 bytes 2024-09-11T05:59:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1632022528 bytes 2024-09-11T05:59:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1645553664 bytes 2024-09-11T05:59:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1700449280 bytes 2024-09-11T05:59:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1747783680 bytes 2024-09-11T05:59:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1795976192 bytes 2024-09-11T05:59:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1843326976 bytes 2024-09-11T05:59:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1864654848 bytes 2024-09-11T05:59:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1907699712 bytes 2024-09-11T05:59:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1952510976 bytes 2024-09-11T05:59:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1996617728 bytes 2024-09-11T05:59:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2029003776 bytes 2024-09-11T05:59:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2051064832 bytes 2024-09-11T05:59:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2068967424 bytes 2024-09-11T05:59:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2094789632 bytes 2024-09-11T05:59:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2167598080 bytes 2024-09-11T05:59:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2221160448 bytes 2024-09-11T05:59:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2259802112 bytes 2024-09-11T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2282381312 bytes 2024-09-11T05:59:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2326933504 bytes 2024-09-11T05:59:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2405308416 bytes 2024-09-11T05:59:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2423046144 bytes 2024-09-11T05:59:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2489783296 bytes 2024-09-11T05:59:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2533866496 bytes 2024-09-11T05:59:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2573536256 bytes 2024-09-11T05:59:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2612913152 bytes 2024-09-11T05:59:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2654801920 bytes 2024-09-11T05:59:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2685105152 bytes 2024-09-11T05:59:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2760076288 bytes 2024-09-11T05:59:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2802449408 bytes 2024-09-11T05:59:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2824274944 bytes 2024-09-11T05:59:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2854553600 bytes 2024-09-11T05:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2917828608 bytes 2024-09-11T05:59:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2956607488 bytes 2024-09-11T05:59:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3057511424 bytes 2024-09-11T05:59:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3095910400 bytes 2024-09-11T05:59:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3165671424 bytes 2024-09-11T05:59:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3240460288 bytes 2024-09-11T05:59:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3274996736 bytes 2024-09-11T05:59:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3293813760 bytes 2024-09-11T05:59:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3337502720 bytes 2024-09-11T05:59:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3394656256 bytes 2024-09-11T05:59:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3480798208 bytes 2024-09-11T05:59:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3513256960 bytes 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:59:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T05:59:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:59:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T05:59:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120b3c03897a 2024-09-11T05:59:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T05:59:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T05:59:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 present 120b3c03897a 2024-09-11T05:59:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T05:59:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120b3c03897a 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T05:59:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T05:59:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T05:59:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T05:59:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T05:59:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120b3c03897a 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120b3c03897a 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120b3c03897a -o 120b3c03897a = 120b3c03897a 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T05:59:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-11T05:59:36.732 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6418063436427243) > weight (0.25) 2024-09-11T05:59:36.733 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5395457786391216) > weight (0.25) 2024-09-11T05:59:36.733 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-11T05:59:36.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-11T05:59:36.734 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T05:59:36.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T05:59:36.732+0000 7f9b0cd16640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 34708) UID: 1000 2024-09-11T05:59:39.734 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-11T05:59:39.734 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T05:59:39.735 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-11T05:59:39.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T05:59:39.735 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T05:59:39.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T05:59:39.738 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-11T05:59:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:59:43.880+0000 7fe14da39640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f136b040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T05:59:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T05:59:43.880+0000 7fe153a45640 -1 rbd::mirror::ImageReplayer: 0x5624f1810780 [3/f1f69ba8-2492-401d-abb0-04773b8fbe74] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:00:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:00:02.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:00:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:00:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:00:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T06:00:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:00:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:00:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T06:00:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:00:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:00:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_5 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:00:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T06:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-11T06:00:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T06:00:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T06:00:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2024-09-11T06:00:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T06:00:02.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T06:00:02.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T06:00:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T06:00:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-11T06:00:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-11T06:00:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T06:00:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-11T06:00:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T06:00:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19a6af827520 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19a6af827520 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 06:00:02 2024 2024-09-11T06:00:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 06:00:02 2024 2024-09-11T06:00:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 06:00:02 2024 2024-09-11T06:00:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T06:00:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T06:00:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 876ff8f7-69e5-43d4-b583-8e252e20997c 2024-09-11T06:00:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T06:00:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T06:00:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:00:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T06:00:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:00:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:00:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 57191424 bytes 2024-09-11T06:00:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 87297024 bytes 2024-09-11T06:00:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 119265280 bytes 2024-09-11T06:00:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 179433472 bytes 2024-09-11T06:00:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 255907840 bytes 2024-09-11T06:00:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 283508736 bytes 2024-09-11T06:00:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 307617792 bytes 2024-09-11T06:00:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 339952640 bytes 2024-09-11T06:00:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 423695360 bytes 2024-09-11T06:00:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 455716864 bytes 2024-09-11T06:00:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 494548992 bytes 2024-09-11T06:00:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 517800960 bytes 2024-09-11T06:00:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 609939456 bytes 2024-09-11T06:00:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 620164096 bytes 2024-09-11T06:00:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 662624256 bytes 2024-09-11T06:00:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 698637312 bytes 2024-09-11T06:00:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 750098432 bytes 2024-09-11T06:00:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 775226368 bytes 2024-09-11T06:00:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 836284416 bytes 2024-09-11T06:00:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 857970688 bytes 2024-09-11T06:00:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 917490688 bytes 2024-09-11T06:00:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 997897216 bytes 2024-09-11T06:00:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1025492992 bytes 2024-09-11T06:00:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1092980736 bytes 2024-09-11T06:00:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1120553984 bytes 2024-09-11T06:00:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1166748672 bytes 2024-09-11T06:00:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1232521216 bytes 2024-09-11T06:00:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1316387840 bytes 2024-09-11T06:00:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1327632384 bytes 2024-09-11T06:00:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1377930240 bytes 2024-09-11T06:00:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1417472000 bytes 2024-09-11T06:00:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1491222528 bytes 2024-09-11T06:00:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1518599168 bytes 2024-09-11T06:00:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1539912704 bytes 2024-09-11T06:00:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1599637504 bytes 2024-09-11T06:00:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1664634880 bytes 2024-09-11T06:00:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1740185600 bytes 2024-09-11T06:00:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1780091904 bytes 2024-09-11T06:00:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1900319744 bytes 2024-09-11T06:00:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1943672832 bytes 2024-09-11T06:00:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1981799424 bytes 2024-09-11T06:00:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2043011072 bytes 2024-09-11T06:00:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2076717056 bytes 2024-09-11T06:00:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2133981184 bytes 2024-09-11T06:00:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2191245312 bytes 2024-09-11T06:00:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2218311680 bytes 2024-09-11T06:00:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2266612736 bytes 2024-09-11T06:00:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2300494848 bytes 2024-09-11T06:00:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2301095936 bytes 2024-09-11T06:00:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2361482240 bytes 2024-09-11T06:00:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2400997376 bytes 2024-09-11T06:00:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2425190400 bytes 2024-09-11T06:00:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2460274688 bytes 2024-09-11T06:00:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2513409024 bytes 2024-09-11T06:00:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2535909376 bytes 2024-09-11T06:00:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2608800768 bytes 2024-09-11T06:00:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2690994176 bytes 2024-09-11T06:00:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2736268288 bytes 2024-09-11T06:00:11.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2776079360 bytes 2024-09-11T06:00:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2817267712 bytes 2024-09-11T06:00:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2849735680 bytes 2024-09-11T06:00:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2887213056 bytes 2024-09-11T06:00:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2927521792 bytes 2024-09-11T06:00:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2928135168 bytes 2024-09-11T06:00:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2993483776 bytes 2024-09-11T06:00:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3034524672 bytes 2024-09-11T06:00:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3079763968 bytes 2024-09-11T06:00:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3080374272 bytes 2024-09-11T06:00:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3099265024 bytes 2024-09-11T06:00:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3103932416 bytes 2024-09-11T06:00:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3205452800 bytes 2024-09-11T06:00:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3265415168 bytes 2024-09-11T06:00:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3266022400 bytes 2024-09-11T06:00:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3289159680 bytes 2024-09-11T06:00:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3389036544 bytes 2024-09-11T06:00:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3394191360 bytes 2024-09-11T06:00:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3394805760 bytes 2024-09-11T06:00:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3467180032 bytes 2024-09-11T06:00:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3529403392 bytes 2024-09-11T06:00:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3591659520 bytes 2024-09-11T06:00:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3609119744 bytes 2024-09-11T06:00:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3635027968 bytes 2024-09-11T06:00:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3690440704 bytes 2024-09-11T06:00:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3733998592 bytes 2024-09-11T06:00:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3774365696 bytes 2024-09-11T06:00:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3865351168 bytes 2024-09-11T06:00:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3923909632 bytes 2024-09-11T06:00:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3956089856 bytes 2024-09-11T06:00:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3987261440 bytes 2024-09-11T06:00:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 4034072576 bytes 2024-09-11T06:00:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 4060124160 bytes 2024-09-11T06:00:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 4084593664 bytes 2024-09-11T06:00:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 4160411648 bytes 2024-09-11T06:00:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 4176523264 bytes 2024-09-11T06:00:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 4191167488 bytes 2024-09-11T06:00:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 4259527680 bytes 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:00:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120bdee59672 2024-09-11T06:00:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:00:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 present 120bdee59672 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120bdee59672 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:00:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:00:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:00:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T06:00:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:00:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120bdee59672 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120bdee59672 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120bdee59672 -o 120bdee59672 = 120bdee59672 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-11T06:00:23.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9229673517270748) > weight (0.25) 2024-09-11T06:00:23.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.90907940444631) > weight (0.25) 2024-09-11T06:00:23.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9638150547079883) > weight (0.25) 2024-09-11T06:00:23.610 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-11T06:00:23.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-11T06:00:23.611 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-11T06:00:23.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:00:23.611+0000 7fe330d71640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 61747) UID: 1000 2024-09-11T06:00:32.612 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-11T06:00:32.613 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T06:00:32.613 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-11T06:00:32.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T06:00:32.614 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T06:00:32.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T06:00:32.617 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-11T06:00:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:00:43.879+0000 7fe14da39640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f136b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:00:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:00:43.879+0000 7fe153a45640 -1 rbd::mirror::ImageReplayer: 0x5624f1810780 [3/876ff8f7-69e5-43d4-b583-8e252e20997c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:00:48.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:00:48.039+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ec291400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-11T06:00:48.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:00:48.039+0000 7f9e785cc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5ec291400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-11T06:00:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:00:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T06:00:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:00:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120bdee59672 2024-09-11T06:00:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:00:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:00:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 deleted 120bdee59672 2024-09-11T06:00:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:00:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120bdee59672 2024-09-11T06:00:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:00:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:00:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:00:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_6 2024-09-11T06:00:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:00:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T06:00:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:00:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:00:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-11T06:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-11T06:00:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:00:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T06:00:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2024-09-11T06:00:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T06:00:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T06:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:00:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-11T06:00:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-11T06:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T06:00:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-11T06:00:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T06:00:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-11T06:00:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2024-09-11T06:00:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T06:00:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T06:00:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T06:00:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19c49ae726b2 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19c49ae726b2 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 06:00:48 2024 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 06:00:48 2024 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 06:00:48 2024 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: ed63a1ae-8b42-4d03-b206-ac8ddce25ecd 2024-09-11T06:00:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T06:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T06:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:00:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2024-09-11T06:00:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:00:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:00:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 53276672 bytes 2024-09-11T06:00:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 132700160 bytes 2024-09-11T06:00:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 173284352 bytes 2024-09-11T06:00:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 195261440 bytes 2024-09-11T06:00:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 213990400 bytes 2024-09-11T06:00:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 219945984 bytes 2024-09-11T06:00:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 244580352 bytes 2024-09-11T06:00:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 298613760 bytes 2024-09-11T06:00:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 301307904 bytes 2024-09-11T06:00:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 374855680 bytes 2024-09-11T06:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 432026624 bytes 2024-09-11T06:00:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 494113792 bytes 2024-09-11T06:00:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 546931712 bytes 2024-09-11T06:00:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 569589760 bytes 2024-09-11T06:00:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 608756736 bytes 2024-09-11T06:00:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 697597952 bytes 2024-09-11T06:00:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 771991552 bytes 2024-09-11T06:00:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 800753664 bytes 2024-09-11T06:00:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 801344512 bytes 2024-09-11T06:00:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 822135808 bytes 2024-09-11T06:00:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 885883904 bytes 2024-09-11T06:00:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 943549440 bytes 2024-09-11T06:00:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 951285760 bytes 2024-09-11T06:00:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1011904512 bytes 2024-09-11T06:00:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1029156864 bytes 2024-09-11T06:00:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1107208192 bytes 2024-09-11T06:00:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1142284288 bytes 2024-09-11T06:00:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1163425792 bytes 2024-09-11T06:00:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1166150656 bytes 2024-09-11T06:00:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1189354496 bytes 2024-09-11T06:00:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1192180736 bytes 2024-09-11T06:00:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1227977728 bytes 2024-09-11T06:00:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1275428864 bytes 2024-09-11T06:00:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1344010240 bytes 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T06:00:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:00:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:00:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:00:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120b30c8c9a4 2024-09-11T06:00:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:00:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 present 120b30c8c9a4 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120b30c8c9a4 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:00:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:00:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:00:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120b30c8c9a4 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120b30c8c9a4 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120b30c8c9a4 -o 120b30c8c9a4 = 120b30c8c9a4 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:00:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-11T06:00:54.739 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T06:00:54.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T06:00:54.740 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T06:00:54.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:00:54.738+0000 7f7f179ff640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 61405) UID: 1000 2024-09-11T06:00:57.741 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T06:00:57.741 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T06:00:57.742 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T06:00:57.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T06:00:57.742 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T06:00:57.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T06:00:57.745 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-11T06:01:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:13.878+0000 7fe14d238640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f189e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:01:13.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:13.878+0000 7fe154246640 -1 rbd::mirror::ImageReplayer: 0x5624efce4000 [3/ed63a1ae-8b42-4d03-b206-ac8ddce25ecd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:01:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:01:25.110+0000 7f9e77dcb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5f0db2800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-11T06:01:25.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:01:25.110+0000 7f9e77dcb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f5f0db2800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-11T06:01:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:01:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T06:01:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:01:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T06:01:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:01:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:01:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T06:01:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T06:01:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:01:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:01:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:01:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:01:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:01:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:01:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:01:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_7 2024-09-11T06:01:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:01:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:01:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T06:01:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:01:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:01:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:01:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:01:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-11T06:01:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:01:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-11T06:01:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T06:01:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:01:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2024-09-11T06:01:25.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T06:01:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T06:01:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:01:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-11T06:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-11T06:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T06:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-11T06:01:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T06:01:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19e55b5f2b6f 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19e55b5f2b6f 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T06:01:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 06:01:25 2024 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 06:01:25 2024 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 06:01:25 2024 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 729e68be-4d40-497e-a3e3-6dfdd47762de 2024-09-11T06:01:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T06:01:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T06:01:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:01:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2024-09-11T06:01:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:01:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:01:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 82982912 bytes 2024-09-11T06:01:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 97442816 bytes 2024-09-11T06:01:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 114933760 bytes 2024-09-11T06:01:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 120564736 bytes 2024-09-11T06:01:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 148512768 bytes 2024-09-11T06:01:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 188313600 bytes 2024-09-11T06:01:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 251355136 bytes 2024-09-11T06:01:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 301883392 bytes 2024-09-11T06:01:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 337303552 bytes 2024-09-11T06:01:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 370150400 bytes 2024-09-11T06:01:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 402659328 bytes 2024-09-11T06:01:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 475576320 bytes 2024-09-11T06:01:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 485499904 bytes 2024-09-11T06:01:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 549041152 bytes 2024-09-11T06:01:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 549644288 bytes 2024-09-11T06:01:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 612371456 bytes 2024-09-11T06:01:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 618061824 bytes 2024-09-11T06:01:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 695025664 bytes 2024-09-11T06:01:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 726135808 bytes 2024-09-11T06:01:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 787181568 bytes 2024-09-11T06:01:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 825623552 bytes 2024-09-11T06:01:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 884726784 bytes 2024-09-11T06:01:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 927952896 bytes 2024-09-11T06:01:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 962393088 bytes 2024-09-11T06:01:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1013617664 bytes 2024-09-11T06:01:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1027072000 bytes 2024-09-11T06:01:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1110571008 bytes 2024-09-11T06:01:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1153183744 bytes 2024-09-11T06:01:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1181159424 bytes 2024-09-11T06:01:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1245785088 bytes 2024-09-11T06:01:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1261352960 bytes 2024-09-11T06:01:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1298812928 bytes 2024-09-11T06:01:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1339978752 bytes 2024-09-11T06:01:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1358521344 bytes 2024-09-11T06:01:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1414636544 bytes 2024-09-11T06:01:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1444684800 bytes 2024-09-11T06:01:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1509240832 bytes 2024-09-11T06:01:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1569295360 bytes 2024-09-11T06:01:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1614353408 bytes 2024-09-11T06:01:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1669890048 bytes 2024-09-11T06:01:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1700352000 bytes 2024-09-11T06:01:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1754036224 bytes 2024-09-11T06:01:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1760967680 bytes 2024-09-11T06:01:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1819701248 bytes 2024-09-11T06:01:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1884442624 bytes 2024-09-11T06:01:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1942732800 bytes 2024-09-11T06:01:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1978025984 bytes 2024-09-11T06:01:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1984382976 bytes 2024-09-11T06:01:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1998996480 bytes 2024-09-11T06:01:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2044480512 bytes 2024-09-11T06:01:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2064778240 bytes 2024-09-11T06:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2098127872 bytes 2024-09-11T06:01:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2152350720 bytes 2024-09-11T06:01:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2192451584 bytes 2024-09-11T06:01:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2272045056 bytes 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:01:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:01:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:01:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:01:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:01:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:01:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T06:01:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:01:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:01:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=120b6c05e492 2024-09-11T06:01:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:01:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:01:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 present 120b6c05e492 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=120b6c05e492 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:01:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:01:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:01:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:01:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:01:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T06:01:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:01:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=120b6c05e492 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 120b6c05e492 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 120b6c05e492 -o 120b6c05e492 = 120b6c05e492 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:01:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-11T06:01:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:01:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:01:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:01:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-11T06:01:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:43.882+0000 7fe14d238640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f189fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:01:43.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:43.882+0000 7fe153a45640 -1 rbd::mirror::ImageReplayer: 0x5624efce4000 [3/729e68be-4d40-497e-a3e3-6dfdd47762de] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:01:50.618 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T06:01:50.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T06:01:50.619 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-11T06:01:50.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:01:50.618+0000 7f9e85de7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 35187) UID: 1000 2024-09-11T06:01:54.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:54.806+0000 7fe14d238640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5624f189fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:01:54.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:01:54.806+0000 7fe153a45640 -1 rbd::mirror::ImageReplayer: 0x5624efce4000 [3/729e68be-4d40-497e-a3e3-6dfdd47762de] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:01:55.620 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T06:01:55.620 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T06:01:55.620 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T06:01:55.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T06:01:55.621 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T06:01:55.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T06:01:55.624 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-11T06:02:04.746 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3815937468617726) > weight (0.25) 2024-09-11T06:02:04.746 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T06:02:04.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T06:02:04.746 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T06:02:04.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:02:04.745+0000 7fe154a47640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45028) UID: 1000 2024-09-11T06:02:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:02:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T06:02:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:02:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:02:06.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:02:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T06:02:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:02:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:02:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T06:02:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:02:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:02:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:02:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:02:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:02:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:02:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_8 2024-09-11T06:02:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:02:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:02:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:02:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-11T06:02:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T06:02:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2024-09-11T06:02:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T06:02:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T06:02:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:02:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T06:02:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T06:02:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T06:02:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-11T06:02:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T06:02:08.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-11T06:02:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19ffaae56eb6 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19ffaae56eb6 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 06:02:06 2024 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 06:02:06 2024 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 06:02:06 2024 2024-09-11T06:02:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T06:02:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T06:02:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7240e98e-a7c3-4c9c-8e8d-3c8a7c2adff6 2024-09-11T06:02:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T06:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T06:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:02:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2024-09-11T06:02:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:02:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:02:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 34208768 bytes 2024-09-11T06:02:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 36017152 bytes 2024-09-11T06:02:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 103547904 bytes 2024-09-11T06:02:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 140469248 bytes 2024-09-11T06:02:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 173637632 bytes 2024-09-11T06:02:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 213950464 bytes 2024-09-11T06:02:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 261728256 bytes 2024-09-11T06:02:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 267199488 bytes 2024-09-11T06:02:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 318908416 bytes 2024-09-11T06:02:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 416357376 bytes 2024-09-11T06:02:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 474433536 bytes 2024-09-11T06:02:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 502360064 bytes 2024-09-11T06:02:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 524361728 bytes 2024-09-11T06:02:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 581642240 bytes 2024-09-11T06:02:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 622468096 bytes 2024-09-11T06:02:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 629623808 bytes 2024-09-11T06:02:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 648612864 bytes 2024-09-11T06:02:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 698565632 bytes 2024-09-11T06:02:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 699179008 bytes 2024-09-11T06:02:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 715227136 bytes 2024-09-11T06:02:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 754618368 bytes 2024-09-11T06:02:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 776973312 bytes 2024-09-11T06:02:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 813304832 bytes 2024-09-11T06:02:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 901740544 bytes 2024-09-11T06:02:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 961772544 bytes 2024-09-11T06:02:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1123996672 bytes 2024-09-11T06:02:12.747 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T06:02:12.747 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T06:02:12.748 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T06:02:12.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T06:02:12.748 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T06:02:12.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T06:02:12.751 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-11T06:02:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1184714752 bytes 2024-09-11T06:02:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1225143296 bytes 2024-09-11T06:02:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1246641152 bytes 2024-09-11T06:02:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1254960128 bytes 2024-09-11T06:02:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1255581696 bytes 2024-09-11T06:02:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1287065600 bytes 2024-09-11T06:02:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1289029632 bytes 2024-09-11T06:02:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1292906496 bytes 2024-09-11T06:02:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1353129984 bytes 2024-09-11T06:02:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1360876544 bytes 2024-09-11T06:02:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1383389184 bytes 2024-09-11T06:02:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1438807040 bytes 2024-09-11T06:02:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1469493248 bytes 2024-09-11T06:02:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1496602624 bytes 2024-09-11T06:02:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1500624896 bytes 2024-09-11T06:02:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1606801408 bytes 2024-09-11T06:02:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1634786304 bytes 2024-09-11T06:02:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1726050304 bytes 2024-09-11T06:02:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1763134464 bytes 2024-09-11T06:02:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1782484992 bytes 2024-09-11T06:02:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1847511040 bytes 2024-09-11T06:02:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1885550592 bytes 2024-09-11T06:02:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1917580288 bytes 2024-09-11T06:02:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1947462656 bytes 2024-09-11T06:02:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1982358528 bytes 2024-09-11T06:02:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1984279552 bytes 2024-09-11T06:02:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2075287552 bytes 2024-09-11T06:02:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2109478912 bytes 2024-09-11T06:02:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2140697600 bytes 2024-09-11T06:02:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2207532032 bytes 2024-09-11T06:02:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2246031360 bytes 2024-09-11T06:02:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2278311936 bytes 2024-09-11T06:02:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2312210432 bytes 2024-09-11T06:02:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2345768960 bytes 2024-09-11T06:02:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2372880384 bytes 2024-09-11T06:02:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2391631872 bytes 2024-09-11T06:02:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2462860288 bytes 2024-09-11T06:02:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2519182336 bytes 2024-09-11T06:02:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2554809344 bytes 2024-09-11T06:02:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2589903872 bytes 2024-09-11T06:02:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2618801152 bytes 2024-09-11T06:02:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2678575104 bytes 2024-09-11T06:02:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2694891520 bytes 2024-09-11T06:02:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2738500608 bytes 2024-09-11T06:02:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2812823552 bytes 2024-09-11T06:02:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2826527744 bytes 2024-09-11T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2869139456 bytes 2024-09-11T06:02:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2930054144 bytes 2024-09-11T06:02:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2947032064 bytes 2024-09-11T06:02:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2947696640 bytes 2024-09-11T06:02:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3013900288 bytes 2024-09-11T06:02:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3062394880 bytes 2024-09-11T06:02:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3090529280 bytes 2024-09-11T06:02:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3142657024 bytes 2024-09-11T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3148337152 bytes 2024-09-11T06:02:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3204171776 bytes 2024-09-11T06:02:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3236416512 bytes 2024-09-11T06:02:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3306370048 bytes 2024-09-11T06:02:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8500 IOs, 3320660992 bytes 2024-09-11T06:02:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8600 IOs, 3332861952 bytes 2024-09-11T06:02:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8700 IOs, 3342788608 bytes 2024-09-11T06:02:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8800 IOs, 3355379712 bytes 2024-09-11T06:02:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8900 IOs, 3377859584 bytes 2024-09-11T06:02:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9000 IOs, 3392566272 bytes 2024-09-11T06:02:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9100 IOs, 3471662080 bytes 2024-09-11T06:02:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9200 IOs, 3479865344 bytes 2024-09-11T06:02:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9300 IOs, 3493160960 bytes 2024-09-11T06:02:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9400 IOs, 3548438528 bytes 2024-09-11T06:02:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9500 IOs, 3575918592 bytes 2024-09-11T06:02:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9600 IOs, 3608598528 bytes 2024-09-11T06:02:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9700 IOs, 3673659392 bytes 2024-09-11T06:02:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:9800 IOs, 3699634176 bytes 2024-09-11T06:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T06:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:02:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:02:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:02:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=1791bfdf0933 2024-09-11T06:02:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:02:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 present 1791bfdf0933 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=1791bfdf0933 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:02:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T06:02:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:02:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=1791bfdf0933 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1791bfdf0933 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 1791bfdf0933 -o 1791bfdf0933 = 1791bfdf0933 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-11T06:02:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-11T06:02:27.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:02:27.941+0000 7f61312b1640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574e3743a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:02:27.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:02:27.941+0000 7f61372bd640 -1 rbd::mirror::ImageReplayer: 0x5574e3296780 [3/7240e98e-a7c3-4c9c-8e8d-3c8a7c2adff6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:02:53.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:02:53.145+0000 7f0e711b1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e4b006d400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-11T06:02:53.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:02:53.145+0000 7f0e711b1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e4b006d400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-11T06:02:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:02:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T06:02:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:02:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:02:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T06:02:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:02:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:02:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:02:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:02:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:02:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:02:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:02:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_9 2024-09-11T06:02:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:02:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:02:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-11T06:02:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-11T06:02:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T06:02:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2024-09-11T06:02:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T06:02:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T06:02:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for i in `seq 1 10` 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -n 512M ']' 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2024-09-11T06:02:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2024-09-11T06:02:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-11T06:02:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2024-09-11T06:02:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1a1c284b858b 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1a1c284b858b 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2024-09-11T06:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Wed Sep 11 06:02:53 2024 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Wed Sep 11 06:02:53 2024 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Wed Sep 11 06:02:53 2024 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 36a0a71d-6bd1-4476-b4be-d786c72571a5 2024-09-11T06:02:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2024-09-11T06:02:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T06:02:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:02:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:02:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:02:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:02:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2024-09-11T06:02:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:02:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:02:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 103097344 bytes 2024-09-11T06:02:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 124874752 bytes 2024-09-11T06:02:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 175021056 bytes 2024-09-11T06:02:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 195862528 bytes 2024-09-11T06:02:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 218746880 bytes 2024-09-11T06:02:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 249673728 bytes 2024-09-11T06:02:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 273080320 bytes 2024-09-11T06:02:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 297309184 bytes 2024-09-11T06:02:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 353659904 bytes 2024-09-11T06:02:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 378200064 bytes 2024-09-11T06:02:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 406028288 bytes 2024-09-11T06:02:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 406618112 bytes 2024-09-11T06:02:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 490658816 bytes 2024-09-11T06:02:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 521051136 bytes 2024-09-11T06:02:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 553254912 bytes 2024-09-11T06:02:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 639078400 bytes 2024-09-11T06:02:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 734519296 bytes 2024-09-11T06:02:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 826604544 bytes 2024-09-11T06:02:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 908633088 bytes 2024-09-11T06:02:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 936767488 bytes 2024-09-11T06:02:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 966582272 bytes 2024-09-11T06:02:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1029443584 bytes 2024-09-11T06:02:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1052859392 bytes 2024-09-11T06:02:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1130015744 bytes 2024-09-11T06:02:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1158247424 bytes 2024-09-11T06:02:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1196303360 bytes 2024-09-11T06:02:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1196913664 bytes 2024-09-11T06:02:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1268362240 bytes 2024-09-11T06:02:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1306261504 bytes 2024-09-11T06:02:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1409648640 bytes 2024-09-11T06:02:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1429371904 bytes 2024-09-11T06:02:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1472543744 bytes 2024-09-11T06:02:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1494761472 bytes 2024-09-11T06:02:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1556426752 bytes 2024-09-11T06:02:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1621671936 bytes 2024-09-11T06:02:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1665059840 bytes 2024-09-11T06:02:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1718832128 bytes 2024-09-11T06:02:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1719428096 bytes 2024-09-11T06:03:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1754597376 bytes 2024-09-11T06:03:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1808774144 bytes 2024-09-11T06:03:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1879573504 bytes 2024-09-11T06:03:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1923523584 bytes 2024-09-11T06:03:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1945116672 bytes 2024-09-11T06:03:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2003418112 bytes 2024-09-11T06:03:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2027661312 bytes 2024-09-11T06:03:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2080373760 bytes 2024-09-11T06:03:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2147881984 bytes 2024-09-11T06:03:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2215962624 bytes 2024-09-11T06:03:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2257080320 bytes 2024-09-11T06:03:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2288503808 bytes 2024-09-11T06:03:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2321225728 bytes 2024-09-11T06:03:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2333550592 bytes 2024-09-11T06:03:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2355719168 bytes 2024-09-11T06:03:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2401287168 bytes 2024-09-11T06:03:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2454885376 bytes 2024-09-11T06:03:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2494129152 bytes 2024-09-11T06:03:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2510861312 bytes 2024-09-11T06:03:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2568698880 bytes 2024-09-11T06:03:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2635513856 bytes 2024-09-11T06:03:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2663497728 bytes 2024-09-11T06:03:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2664105984 bytes 2024-09-11T06:03:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2691884032 bytes 2024-09-11T06:03:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2725394432 bytes 2024-09-11T06:03:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2761151488 bytes 2024-09-11T06:03:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2771154944 bytes 2024-09-11T06:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2808935424 bytes 2024-09-11T06:03:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2823495680 bytes 2024-09-11T06:03:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2845480960 bytes 2024-09-11T06:03:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2874340352 bytes 2024-09-11T06:03:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2936008704 bytes 2024-09-11T06:03:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3047259136 bytes 2024-09-11T06:03:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3064317952 bytes 2024-09-11T06:03:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3064914944 bytes 2024-09-11T06:03:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3119826944 bytes 2024-09-11T06:03:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3143757824 bytes 2024-09-11T06:03:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3152809984 bytes 2024-09-11T06:03:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3177842688 bytes 2024-09-11T06:03:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3180124160 bytes 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:03:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:03:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=17913f668377 2024-09-11T06:03:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:03:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 present 17913f668377 2024-09-11T06:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17913f668377 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:03:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T06:03:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:03:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:03:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17913f668377 2024-09-11T06:03:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17913f668377 2024-09-11T06:03:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17913f668377 -o 17913f668377 = 17913f668377 2024-09-11T06:03:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2024-09-11T06:03:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-11T06:03:27.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:27.940+0000 7f6130ab0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574e3743860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T06:03:27.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:27.940+0000 7f61372bd640 -1 rbd::mirror::ImageReplayer: 0x5574e3296780 [3/36a0a71d-6bd1-4476-b4be-d786c72571a5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T06:03:36.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:36.140+0000 7f0e711b1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e4b01d2800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-11T06:03:36.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:36.140+0000 7f0e711b1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e4b01d2800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=75c5b592-5745-4d1a-984f-ff4a3feb0219, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-11T06:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2024-09-11T06:03:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T06:03:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:03:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:03:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:03:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2024-09-11T06:03:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:03:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2024-09-11T06:03:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T06:03:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2024-09-11T06:03:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2024-09-11T06:03:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2024-09-11T06:03:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2024-09-11T06:03:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:03:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local image=image_10 2024-09-11T06:03:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:03:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:03:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:03:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2024-09-11T06:03:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n '' 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:03:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T06:03:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2024-09-11T06:03:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T06:03:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T06:03:36.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5508275223060196) > weight (0.25) 2024-09-11T06:03:36.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6400603898793962) > weight (0.25) 2024-09-11T06:03:36.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.602224905182286) > weight (0.25) 2024-09-11T06:03:36.751 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4536688463819447) > weight (0.25) 2024-09-11T06:03:36.752 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-11T06:03:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:03:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-11T06:03:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.Ange9SOiZn/rbd-mirror.test.log 2024-09-11T06:03:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ date '+%F %T' 2024-09-11T06:03:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2024-09-11 06:03:37 TEST: check if removed 'images'\''' OMAP are removed 2024-09-11T06:03:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2024-09-11 06:03:37 TEST: check if removed images' OMAP are removed 2024-09-11T06:03:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-11T06:03:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-11T06:03:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:03:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2024-09-11T06:03:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-11T06:03:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=status_global 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:03:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c status_global 2024-09-11T06:03:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=image_ 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:03:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c image_ 2024-09-11T06:03:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T06:03:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2024-09-11T06:03:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=image_map 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c image_map 2024-09-11T06:03:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T06:03:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 = 0 2024-09-11T06:03:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=1 2024-09-11T06:03:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 1 -eq 0 ']' 2024-09-11T06:03:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2024-09-11T06:03:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-11T06:03:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=image_map 2024-09-11T06:03:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-11T06:03:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster1 2024-09-11T06:03:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c image_map 2024-09-11T06:03:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=status_global 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c status_global 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-11T06:03:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:03:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2024-09-11T06:03:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=image_ 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:03:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:03:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c image_ 2024-09-11T06:03:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2024-09-11T06:03:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-11T06:03:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local filter=image_map 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local cluster=cluster2 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local pool=mirror 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T06:03:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ grep -c image_map 2024-09-11T06:03:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2024-09-11T06:03:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2024-09-11T06:03:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2024-09-11T06:03:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cleanup 0 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code=0 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -ne 0 ']' 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z '' ']' 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster instance 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T06:03:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T06:03:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' does not exist 2024-09-11T06:03:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T06:03:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T06:03:40.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.180+0000 7f6ffd6cd640 -1 rbd::mirror::RemotePoolPoller: 0x5575b52c6340 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-11T06:03:40.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.180+0000 7f6ffd6cd640 -1 rbd::mirror::RemotePoolPoller: 0x5575b52c6340 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-11T06:03:40.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.180+0000 7f6ffd6cd640 -1 rbd::mirror::PoolReplayer: 0x5575b52d7b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-11T06:03:40.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.452+0000 7f700df2d880 -1 rbd::mirror::Mirror: 0x5575b5565e60 update_pool_replayers: restarting failed pool replayer for uuid: f47c6719-5099-4349-a7c0-e7564c6a2880 cluster: cluster1 client: client.mirror 2024-09-11T06:03:40.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.480+0000 7f6ffcecc640 -1 rbd::mirror::MirrorStatusUpdater 0x5575b6e606c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:40.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.480+0000 7f700a6e7640 -1 rbd::mirror::NamespaceReplayer: 0x5575b61821a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:40.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:40.540+0000 7f700df2d880 -1 rbd::mirror::PoolReplayer: 0x5575b52d7b00 init: error accessing remote pool mirror: (2) No such file or directory 2024-09-11T06:03:40.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.828+0000 7f0e7e1cb640 -1 rbd::mirror::LeaderWatcher: 0x55e4b08b6c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T06:03:40.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.828+0000 7f0e781bf640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e4afedd440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:40.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.828+0000 7f0e781bf640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e4afbf2000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:40.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.828+0000 7f0e781bf640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e4b08a4480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:40.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.832+0000 7f0e779be640 -1 rbd::mirror::PoolWatcher: 0x55e4b00e01e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:40.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.832+0000 7f0e779be640 -1 rbd::mirror::PoolWatcher: 0x55e4b00e03c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:40.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.832+0000 7f0e779be640 -1 rbd::mirror::PoolWatcher: 0x55e4b006fc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:40.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.840+0000 7f0e7e9cc640 -1 rbd::mirror::LeaderWatcher: 0x55e4b08b6c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T06:03:40.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:40.840+0000 7f0e7e9cc640 -1 rbd::mirror::LeaderWatcher: 0x55e4b08b6c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T06:03:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' does not exist 2024-09-11T06:03:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T06:03:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T06:03:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' does not exist 2024-09-11T06:03:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= 2024-09-11T06:03:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T06:03:42.625 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T06:03:42.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T06:03:42.626 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T06:03:42.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.624+0000 7fd3716ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 63184) UID: 1000 2024-09-11T06:03:42.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.624+0000 7fd370eb9640 -1 rbd::mirror::LeaderWatcher: 0x5619032cd200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:42.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd3706b8640 -1 rbd::mirror::InstanceWatcher: 0x561903578000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6161 object: (2) No such file or directory 2024-09-11T06:03:42.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd369eab640 -1 rbd::mirror::InstanceWatcher: 0x5619030a6380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:42.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd3706b8640 -1 rbd::mirror::InstanceWatcher: 0x5619030a7c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6164 object: (2) No such file or directory 2024-09-11T06:03:42.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd36a6ac640 -1 rbd::mirror::InstanceWatcher: 0x561903578000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:42.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd363e9f640 -1 rbd::mirror::MirrorStatusUpdater 0x5619031185a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:42.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd3706b8640 -1 rbd::mirror::NamespaceReplayer: 0x561903116000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd36369e640 -1 rbd::mirror::MirrorStatusUpdater 0x561903118360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd3706b8640 -1 rbd::mirror::NamespaceReplayer: 0x5619031161a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd35c690640 -1 rbd::mirror::InstanceWatcher: 0x5619030a7c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd369eab640 -1 rbd::mirror::MirrorStatusUpdater 0x561903118120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:42.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.628+0000 7fd370eb9640 -1 rbd::mirror::NamespaceReplayer: 0x5619031161a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:42.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.632+0000 7fd35ce91640 -1 rbd::mirror::MirrorStatusUpdater 0x5619032aaea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:42.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.632+0000 7fd370eb9640 -1 rbd::mirror::NamespaceReplayer: 0x5619031164e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:42.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.636+0000 7fd370eb9640 -1 rbd::mirror::InstanceWatcher: 0x5619030a6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6161 object: (2) No such file or directory 2024-09-11T06:03:42.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.636+0000 7fd369eab640 -1 rbd::mirror::InstanceWatcher: 0x5619030a6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:42.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.636+0000 7fd363e9f640 -1 rbd::mirror::MirrorStatusUpdater 0x561903d086c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:42.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:42.636+0000 7fd3706b8640 -1 rbd::mirror::NamespaceReplayer: 0x5619030341a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.052+0000 7fc76cbda640 -1 rbd::mirror::RemotePoolPoller: 0x55798b07ea90 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-11T06:03:43.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.052+0000 7fc76d3db640 -1 rbd::mirror::RemotePoolPoller: 0x55798b07ea90 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-11T06:03:43.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.052+0000 7fc76d3db640 -1 rbd::mirror::PoolReplayer: 0x55798b08fb00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-11T06:03:43.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc77a3f5640 -1 rbd::mirror::InstanceWatcher: 0x55798c1ee380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6670 object: (2) No such file or directory 2024-09-11T06:03:43.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc77a3f5640 -1 rbd::mirror::InstanceWatcher: 0x55798c1eea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6670 object: (2) No such file or directory 2024-09-11T06:03:43.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc773be8640 -1 rbd::mirror::InstanceWatcher: 0x55798c1ee380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:43.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc773be8640 -1 rbd::mirror::InstanceWatcher: 0x55798c1eea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:43.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc76cbda640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c25c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc76cbda640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c25c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc779bf4640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc779bf4640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc7733e7640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c25c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc7733e7640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c25c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc77a3f5640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.056+0000 7fc77a3f5640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' does not exist 2024-09-11T06:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T06:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirrors cluster1 2024-09-11T06:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2024-09-11T06:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 0 3 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:0 '' 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:1 '' 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:2 '' 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:3 '' 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirrors cluster2 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:++ seq 0 3 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:0 '' 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:0 2024-09-11T06:03:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:1 '' 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:1 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:2 '' 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:2 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:3 '' 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:3 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' -z 1 ']' 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test '' = /tmp/tmp.Ange9SOiZn 2024-09-11T06:03:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -Rf /tmp/tmp.Ange9SOiZn 2024-09-11T06:03:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:03:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo OK 2024-09-11T06:03:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:OK 2024-09-11T06:03:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ exit 0 2024-09-11T06:03:43.287 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T06:03:43.287 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-11T06:03:43.449 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-11T06:03:43.449 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-11T06:03:43.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.492+0000 7fc75fbc0640 -1 rbd::mirror::RemotePoolPoller: 0x55798b07fa00 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-11T06:03:43.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.492+0000 7fc75f3bf640 -1 rbd::mirror::RemotePoolPoller: 0x55798b07fa00 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-11T06:03:43.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.492+0000 7fc75f3bf640 -1 rbd::mirror::PoolReplayer: 0x55798c134f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-11T06:03:43.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc779bf4640 -1 rbd::mirror::LeaderWatcher: 0x55798c134c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc779bf4640 -1 rbd::mirror::InstanceWatcher: 0x55798c1ee000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6670 object: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc773be8640 -1 rbd::mirror::InstanceWatcher: 0x55798c1ee000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc76d3db640 -1 rbd::mirror::MirrorStatusUpdater 0x55798cc246c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc77a3f5640 -1 rbd::mirror::NamespaceReplayer: 0x55798bf50000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc7733e7640 -1 rbd::mirror::MirrorStatusUpdater 0x55798cbbefc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.572+0000 7fc779bf4640 -1 rbd::mirror::NamespaceReplayer: 0x55798bf50000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.576+0000 7fc7663cd640 -1 rbd::mirror::InstanceWatcher: 0x55798c6ec000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.576+0000 7fc75fbc0640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c46a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.576+0000 7fc77a3f5640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.580+0000 7fc765bcc640 -1 rbd::mirror::MirrorStatusUpdater 0x55798c3ff0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:43.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:43.580+0000 7fc77a3f5640 -1 rbd::mirror::NamespaceReplayer: 0x55798c25a4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:43.715 DEBUG:teuthology.parallel:result is None 2024-09-11T06:03:43.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:03:43.767 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:03:43.767 DEBUG:teuthology.orchestra.run.smithi164:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T06:03:43.813 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:03:43.813 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-11T06:03:43.827 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-11T06:03:43.828 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2024-09-11T06:03:43.828 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-11T06:03:43.828 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-11T06:03:43.837 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-11T06:03:43.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x56484275f200 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T06:03:43.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd928045640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56484247a480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:43.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd928846640 -1 rbd::mirror::PoolWatcher: 0x564842b2bc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:43.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd922039640 -1 rbd::mirror::PoolWatcher: 0x564842c1cb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:43.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x56484275f200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T06:03:43.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:43.944+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x56484275f200 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T06:03:44.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.916+0000 7f700a6e7640 -1 rbd::mirror::LeaderWatcher: 0x5575b664d200 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T06:03:44.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.916+0000 7f6ff5ebe640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5575b658e000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:44.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.916+0000 7f6ff66bf640 -1 rbd::mirror::PoolWatcher: 0x5575b6372780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.916+0000 7f6fefeb2640 -1 rbd::mirror::PoolWatcher: 0x5575b6372b40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.920+0000 7f700a6e7640 -1 rbd::mirror::LeaderWatcher: 0x5575b664d200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T06:03:44.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:44.920+0000 7f700a6e7640 -1 rbd::mirror::LeaderWatcher: 0x5575b664d200 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T06:03:44.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f6137abe640 -1 rbd::mirror::LeaderWatcher: 0x5574e2c68c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T06:03:44.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f6130ab0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5574e312f680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:44.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f6130ab0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5574e2c72480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:44.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f6130ab0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5574e312e000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f61312b1640 -1 rbd::mirror::PoolWatcher: 0x5574e3237c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f6130ab0640 -1 rbd::mirror::PoolWatcher: 0x5574e3237a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f61312b1640 -1 rbd::mirror::PoolWatcher: 0x5574e3237e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f612aaa4640 -1 rbd::mirror::PoolWatcher: 0x5574e32903c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f612aaa4640 -1 rbd::mirror::PoolWatcher: 0x5574e32905a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f612aaa4640 -1 rbd::mirror::PoolWatcher: 0x5574e3290000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T06:03:44.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f61372bd640 -1 rbd::mirror::LeaderWatcher: 0x5574e2c68c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T06:03:44.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:44.968+0000 7f61372bd640 -1 rbd::mirror::LeaderWatcher: 0x5574e2c68c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T06:03:45.627 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T06:03:45.627 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T06:03:45.627 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T06:03:45.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T06:03:45.627 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T06:03:45.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T06:03:45.630 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-09-11T06:03:45.630 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-11T06:03:45.631 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.641 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-11T06:03:45.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.640+0000 7f929d5aa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 60530) UID: 1000 2024-09-11T06:03:45.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.640+0000 7f929cda9640 -1 rbd::mirror::InstanceWatcher: 0x558e48220380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6423 object: (2) No such file or directory 2024-09-11T06:03:45.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.640+0000 7f929cda9640 -1 rbd::mirror::InstanceWatcher: 0x558e48220a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6423 object: (2) No such file or directory 2024-09-11T06:03:45.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f9288580640 -1 rbd::mirror::InstanceWatcher: 0x558e48220700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f9295d9b640 -1 rbd::mirror::InstanceWatcher: 0x558e48220380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f9295d9b640 -1 rbd::mirror::InstanceWatcher: 0x558e48220a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f928fd8f640 -1 rbd::mirror::MirrorStatusUpdater 0x558e4826e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f928fd8f640 -1 rbd::mirror::MirrorStatusUpdater 0x558e4826e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929cda9640 -1 rbd::mirror::NamespaceReplayer: 0x558e4826c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929cda9640 -1 rbd::mirror::NamespaceReplayer: 0x558e4826c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929659c640 -1 rbd::mirror::MirrorStatusUpdater 0x558e4826e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929659c640 -1 rbd::mirror::MirrorStatusUpdater 0x558e4826e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929c5a8640 -1 rbd::mirror::NamespaceReplayer: 0x558e4826c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.644+0000 7f929cda9640 -1 rbd::mirror::NamespaceReplayer: 0x558e4826c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f929cda9640 -1 rbd::mirror::LeaderWatcher: 0x558e4816ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f929c5a8640 -1 rbd::mirror::InstanceWatcher: 0x558e48220000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6423 object: (2) No such file or directory 2024-09-11T06:03:45.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f9295d9b640 -1 rbd::mirror::InstanceWatcher: 0x558e48220000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f928f58e640 -1 rbd::mirror::MirrorStatusUpdater 0x558e48c7a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f929c5a8640 -1 rbd::mirror::NamespaceReplayer: 0x558e47fa0340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f929659c640 -1 rbd::mirror::MirrorStatusUpdater 0x558e48c26fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.648+0000 7f929cda9640 -1 rbd::mirror::NamespaceReplayer: 0x558e47fa0340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-11T06:03:45.671 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.681 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-11T06:03:45.681 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f700aee8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 62118) UID: 1000 2024-09-11T06:03:45.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f7009ee6640 -1 rbd::mirror::LeaderWatcher: 0x5575b664d200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f700a6e7640 -1 rbd::mirror::InstanceWatcher: 0x5575b68f8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6554 object: (2) No such file or directory 2024-09-11T06:03:45.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f6ff5ebe640 -1 rbd::mirror::InstanceWatcher: 0x5575b68f8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f6fef6b1640 -1 rbd::mirror::MirrorStatusUpdater 0x5575b66dc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f7009ee6640 -1 rbd::mirror::NamespaceReplayer: 0x5575b68f0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f6ff66bf640 -1 rbd::mirror::MirrorStatusUpdater 0x5575b66cd440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.712+0000 7f700a6e7640 -1 rbd::mirror::NamespaceReplayer: 0x5575b68f0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-11T06:03:45.731 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.740 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-11T06:03:45.740 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.772+0000 7fc77abf6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 63474) UID: 1000 2024-09-11T06:03:45.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-11T06:03:45.787 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.796 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-11T06:03:45.796 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f61382bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 62830) UID: 1000 2024-09-11T06:03:45.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.828+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.828+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f61372bd640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2024-09-11T06:03:45.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f6137abe640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2024-09-11T06:03:45.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f6123295640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f61312b1640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.828+0000 7f6130ab0640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f612a2a3640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e2d7e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f612a2a3640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e2d7e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f6137abe640 -1 rbd::mirror::NamespaceReplayer: 0x5574e2d7c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f6137abe640 -1 rbd::mirror::NamespaceReplayer: 0x5574e2d7c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.832+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.832+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f61312b1640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e2d7e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f61312b1640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e2d7e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f61372bd640 -1 rbd::mirror::NamespaceReplayer: 0x5574e2d7c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f61372bd640 -1 rbd::mirror::NamespaceReplayer: 0x5574e2d7c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.832+0000 7f61372bd640 -1 rbd::mirror::LeaderWatcher: 0x5574e2c68c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f61372bd640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f6130ab0640 -1 rbd::mirror::InstanceWatcher: 0x5574e2d10000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f612aaa4640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e37786c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f61372bd640 -1 rbd::mirror::NamespaceReplayer: 0x5574e3636000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f61312b1640 -1 rbd::mirror::MirrorStatusUpdater 0x5574e372afc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f6137abe640 -1 rbd::mirror::NamespaceReplayer: 0x5574e3636000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.836+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.836+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.840+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.840+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.844+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.844+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.848+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.848+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.852+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.852+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.856+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-11T06:03:45.860 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.870 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-11T06:03:45.871 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.856+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.860+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.860+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.864+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.864+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.868+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b6cb2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 62486) UID: 1000 2024-09-11T06:03:45.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.868+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6122 object: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6122 object: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0cea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6125 object: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86af4a3640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86af4a3640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86a1c88640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0cea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86a9497640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cb11e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::NamespaceReplayer: 0x5601cb11c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86a8c96640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cb11e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b64b1640 -1 rbd::mirror::NamespaceReplayer: 0x5601cb11c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86afca4640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cb11e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::NamespaceReplayer: 0x5601cb11c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86af4a3640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cb11e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::NamespaceReplayer: 0x5601cb11c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.872+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.872+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86b5cb0640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86b64b1640 -1 rbd::mirror::LeaderWatcher: 0x5601cb02ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86b64b1640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6122 object: (2) No such file or directory 2024-09-11T06:03:45.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86afca4640 -1 rbd::mirror::InstanceWatcher: 0x5601cb0ce000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86a9497640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cbb226c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86b64b1640 -1 rbd::mirror::NamespaceReplayer: 0x5601cae3e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.876+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86af4a3640 -1 rbd::mirror::MirrorStatusUpdater 0x5601cbaa50e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2024-09-11T06:03:45.876+0000 7f86b5cb0640 -1 rbd::mirror::NamespaceReplayer: 0x5601cae3e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.880+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.884+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.884+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.888+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.892+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.896+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-11T06:03:45.902 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.913 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-11T06:03:45.913 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.900+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.904+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.908+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93d06f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 50666) UID: 1000 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x56484275f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c86e640 -1 rbd::mirror::InstanceWatcher: 0x564842526380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5410 object: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c86e640 -1 rbd::mirror::InstanceWatcher: 0x564842526a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5410 object: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd935860640 -1 rbd::mirror::InstanceWatcher: 0x564842526380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c86e640 -1 rbd::mirror::InstanceWatcher: 0x564842a00000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5413 object: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd936061640 -1 rbd::mirror::InstanceWatcher: 0x564842526a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd92f854640 -1 rbd::mirror::MirrorStatusUpdater 0x564842570360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd92f854640 -1 rbd::mirror::MirrorStatusUpdater 0x5648425705a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd928045640 -1 rbd::mirror::InstanceWatcher: 0x564842a00000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x56484256e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.912+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x56484256e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd935860640 -1 rbd::mirror::MirrorStatusUpdater 0x564842570000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd935860640 -1 rbd::mirror::MirrorStatusUpdater 0x564842570120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x56484256e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x56484256e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd921838640 -1 rbd::mirror::MirrorStatusUpdater 0x5648427847e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x5648429f8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd928846640 -1 rbd::mirror::MirrorStatusUpdater 0x56484273d200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c86e640 -1 rbd::mirror::NamespaceReplayer: 0x5648429f8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.916+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-09-11T06:03:45.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd93c06d640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-09-11T06:03:45.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd93c86e640 -1 rbd::mirror::LeaderWatcher: 0x564842488c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd93c06d640 -1 rbd::mirror::InstanceWatcher: 0x564842526000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5410 object: (2) No such file or directory 2024-09-11T06:03:45.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd935860640 -1 rbd::mirror::InstanceWatcher: 0x564842526000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd92f053640 -1 rbd::mirror::MirrorStatusUpdater 0x564842f806c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd93c86e640 -1 rbd::mirror::NamespaceReplayer: 0x5648422981a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd936061640 -1 rbd::mirror::MirrorStatusUpdater 0x564842f030e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2024-09-11T06:03:45.920+0000 7fd93c06d640 -1 rbd::mirror::NamespaceReplayer: 0x5648422981a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-11T06:03:45.940 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:45.951 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-11T06:03:45.951 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:45.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e7f1cd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 60621) UID: 1000 2024-09-11T06:03:45.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e7e9cc640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6033 object: (2) No such file or directory 2024-09-11T06:03:45.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e7e9cc640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6033 object: (2) No such file or directory 2024-09-11T06:03:45.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e6a1a3640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e781bf640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e781bf640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e719b2640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4afb06360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e719b2640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4afb065a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e7e9cc640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afb04000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.984+0000 7f0e7e9cc640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afb041a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e779be640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4afb06000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e779be640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4afb06120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e7e1cb640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afb04000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e7e1cb640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afb041a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e7e9cc640 -1 rbd::mirror::LeaderWatcher: 0x55e4b08b6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T06:03:45.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e7e9cc640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6033 object: (2) No such file or directory 2024-09-11T06:03:45.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e781bf640 -1 rbd::mirror::InstanceWatcher: 0x55e4afabe000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T06:03:45.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e711b1640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4b07146c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.988+0000 7f0e7e9cc640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afa3a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T06:03:45.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.992+0000 7f0e779be640 -1 rbd::mirror::MirrorStatusUpdater 0x55e4b0698fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T06:03:45.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2024-09-11T06:03:45.992+0000 7f0e7e1cb640 -1 rbd::mirror::NamespaceReplayer: 0x55e4afa3a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T06:03:46.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-11T06:03:46.009 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T06:03:46.020 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-11T06:03:46.020 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:03:46.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2024-09-11T06:03:46.056+0000 7eff4c1f3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64309) UID: 1000 2024-09-11T06:03:46.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-11T06:03:46.070 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T06:03:46.081 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T06:03:46.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:03:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:03:46.414 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T06:03:46.432 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-09-11T06:03:45.999366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3967028,"kb_used_data":17660,"kb_used_omap":4,"kb_used_meta":3949307,"kb_avail":279148492,"statfs":{"total":289910292480,"available":285848055808,"internally_reserved":0,"allocated":18083840,"data_stored":10144308,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4044090703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536786158,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140383,"num_read_kb":-40115589,"num_write":-569655,"num_write_kb":-206078855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572346+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T06:03:43.572346+0000","last_peered":"2024-09-11T06:03:43.572346+0000","last_clean":"2024-09-11T06:03:43.572346+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T06:03:43.572346+0000","last_undegraded":"2024-09-11T06:03:43.572346+0000","last_fullsized":"2024-09-11T06:03:43.572346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.571272+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T06:03:43.571272+0000","last_peered":"2024-09-11T06:03:43.571272+0000","last_clean":"2024-09-11T06:03:43.571272+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T06:03:43.571272+0000","last_undegraded":"2024-09-11T06:03:43.571272+0000","last_fullsized":"2024-09-11T06:03:43.571272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572461+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T06:03:43.572461+0000","last_peered":"2024-09-11T06:03:43.572461+0000","last_clean":"2024-09-11T06:03:43.572461+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T06:03:43.572461+0000","last_undegraded":"2024-09-11T06:03:43.572461+0000","last_fullsized":"2024-09-11T06:03:43.572461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.568538+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T06:03:43.568538+0000","last_peered":"2024-09-11T06:03:43.568538+0000","last_clean":"2024-09-11T06:03:43.568538+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T06:03:43.568538+0000","last_undegraded":"2024-09-11T06:03:43.568538+0000","last_fullsized":"2024-09-11T06:03:43.568538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.277106+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T06:03:43.277106+0000","last_peered":"2024-09-11T06:03:43.277106+0000","last_clean":"2024-09-11T06:03:43.277106+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T06:03:43.277106+0000","last_undegraded":"2024-09-11T06:03:43.277106+0000","last_fullsized":"2024-09-11T06:03:43.277106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:55.262608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.125533+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T06:03:43.125533+0000","last_peered":"2024-09-11T06:03:43.125533+0000","last_clean":"2024-09-11T06:03:43.125533+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T06:03:43.125533+0000","last_undegraded":"2024-09-11T06:03:43.125533+0000","last_fullsized":"2024-09-11T06:03:43.125533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.124622+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T06:03:43.124622+0000","last_peered":"2024-09-11T06:03:43.124622+0000","last_clean":"2024-09-11T06:03:43.124622+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T06:03:43.124622+0000","last_undegraded":"2024-09-11T06:03:43.124622+0000","last_fullsized":"2024-09-11T06:03:43.124622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.569100+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T06:03:43.569100+0000","last_peered":"2024-09-11T06:03:43.569100+0000","last_clean":"2024-09-11T06:03:43.569100+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T06:03:43.569100+0000","last_undegraded":"2024-09-11T06:03:43.569100+0000","last_fullsized":"2024-09-11T06:03:43.569100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1143,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572040+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T06:03:43.572040+0000","last_peered":"2024-09-11T06:03:43.572040+0000","last_clean":"2024-09-11T06:03:43.572040+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T06:03:43.572040+0000","last_undegraded":"2024-09-11T06:03:43.572040+0000","last_fullsized":"2024-09-11T06:03:43.572040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1224940,"kb_used_data":5848,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":93146900,"statfs":{"total":96636764160,"available":95382425600,"internally_reserved":0,"allocated":5988352,"data_stored":2987851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1248328133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273640,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1267710,"kb_avail":93098200,"statfs":{"total":96636764160,"available":95332556800,"internally_reserved":0,"allocated":6049792,"data_stored":3578238,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1298135493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1468448,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":1462526,"kb_avail":92903392,"statfs":{"total":96636764160,"available":95133073408,"internally_reserved":0,"allocated":6045696,"data_stored":3578219,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1497627077},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:03:46.433 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:03:46.722 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:03:46.723 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T06:03:46.740 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-09-11T06:03:45.999366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3967028,"kb_used_data":17660,"kb_used_omap":4,"kb_used_meta":3949307,"kb_avail":279148492,"statfs":{"total":289910292480,"available":285848055808,"internally_reserved":0,"allocated":18083840,"data_stored":10144308,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4044090703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536786158,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140383,"num_read_kb":-40115589,"num_write":-569655,"num_write_kb":-206078855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572346+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T06:03:43.572346+0000","last_peered":"2024-09-11T06:03:43.572346+0000","last_clean":"2024-09-11T06:03:43.572346+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T06:03:43.572346+0000","last_undegraded":"2024-09-11T06:03:43.572346+0000","last_fullsized":"2024-09-11T06:03:43.572346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.571272+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T06:03:43.571272+0000","last_peered":"2024-09-11T06:03:43.571272+0000","last_clean":"2024-09-11T06:03:43.571272+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T06:03:43.571272+0000","last_undegraded":"2024-09-11T06:03:43.571272+0000","last_fullsized":"2024-09-11T06:03:43.571272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572461+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T06:03:43.572461+0000","last_peered":"2024-09-11T06:03:43.572461+0000","last_clean":"2024-09-11T06:03:43.572461+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T06:03:43.572461+0000","last_undegraded":"2024-09-11T06:03:43.572461+0000","last_fullsized":"2024-09-11T06:03:43.572461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.568538+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T06:03:43.568538+0000","last_peered":"2024-09-11T06:03:43.568538+0000","last_clean":"2024-09-11T06:03:43.568538+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T06:03:43.568538+0000","last_undegraded":"2024-09-11T06:03:43.568538+0000","last_fullsized":"2024-09-11T06:03:43.568538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.277106+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T06:03:43.277106+0000","last_peered":"2024-09-11T06:03:43.277106+0000","last_clean":"2024-09-11T06:03:43.277106+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T06:03:43.277106+0000","last_undegraded":"2024-09-11T06:03:43.277106+0000","last_fullsized":"2024-09-11T06:03:43.277106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:55.262608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.125533+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T06:03:43.125533+0000","last_peered":"2024-09-11T06:03:43.125533+0000","last_clean":"2024-09-11T06:03:43.125533+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T06:03:43.125533+0000","last_undegraded":"2024-09-11T06:03:43.125533+0000","last_fullsized":"2024-09-11T06:03:43.125533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.124622+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T06:03:43.124622+0000","last_peered":"2024-09-11T06:03:43.124622+0000","last_clean":"2024-09-11T06:03:43.124622+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T06:03:43.124622+0000","last_undegraded":"2024-09-11T06:03:43.124622+0000","last_fullsized":"2024-09-11T06:03:43.124622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.569100+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T06:03:43.569100+0000","last_peered":"2024-09-11T06:03:43.569100+0000","last_clean":"2024-09-11T06:03:43.569100+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T06:03:43.569100+0000","last_undegraded":"2024-09-11T06:03:43.569100+0000","last_fullsized":"2024-09-11T06:03:43.569100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1143,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572040+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T06:03:43.572040+0000","last_peered":"2024-09-11T06:03:43.572040+0000","last_clean":"2024-09-11T06:03:43.572040+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T06:03:43.572040+0000","last_undegraded":"2024-09-11T06:03:43.572040+0000","last_fullsized":"2024-09-11T06:03:43.572040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1224940,"kb_used_data":5848,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":93146900,"statfs":{"total":96636764160,"available":95382425600,"internally_reserved":0,"allocated":5988352,"data_stored":2987851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1248328133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273640,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1267710,"kb_avail":93098200,"statfs":{"total":96636764160,"available":95332556800,"internally_reserved":0,"allocated":6049792,"data_stored":3578238,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1298135493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1468448,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":1462526,"kb_avail":92903392,"statfs":{"total":96636764160,"available":95133073408,"internally_reserved":0,"allocated":6045696,"data_stored":3578219,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1497627077},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:03:46.741 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T06:03:46.741 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:03:47.023 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:03:47.023 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T06:03:47.040 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-09-11T06:03:45.999366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3967028,"kb_used_data":17660,"kb_used_omap":4,"kb_used_meta":3949307,"kb_avail":279148492,"statfs":{"total":289910292480,"available":285848055808,"internally_reserved":0,"allocated":18083840,"data_stored":10144308,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4044090703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536786158,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140383,"num_read_kb":-40115589,"num_write":-569655,"num_write_kb":-206078855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572346+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T06:03:43.572346+0000","last_peered":"2024-09-11T06:03:43.572346+0000","last_clean":"2024-09-11T06:03:43.572346+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T06:03:43.572346+0000","last_undegraded":"2024-09-11T06:03:43.572346+0000","last_fullsized":"2024-09-11T06:03:43.572346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.571272+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T06:03:43.571272+0000","last_peered":"2024-09-11T06:03:43.571272+0000","last_clean":"2024-09-11T06:03:43.571272+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T06:03:43.571272+0000","last_undegraded":"2024-09-11T06:03:43.571272+0000","last_fullsized":"2024-09-11T06:03:43.571272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572461+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T06:03:43.572461+0000","last_peered":"2024-09-11T06:03:43.572461+0000","last_clean":"2024-09-11T06:03:43.572461+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T06:03:43.572461+0000","last_undegraded":"2024-09-11T06:03:43.572461+0000","last_fullsized":"2024-09-11T06:03:43.572461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.568538+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T06:03:43.568538+0000","last_peered":"2024-09-11T06:03:43.568538+0000","last_clean":"2024-09-11T06:03:43.568538+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T06:03:43.568538+0000","last_undegraded":"2024-09-11T06:03:43.568538+0000","last_fullsized":"2024-09-11T06:03:43.568538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.277106+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T06:03:43.277106+0000","last_peered":"2024-09-11T06:03:43.277106+0000","last_clean":"2024-09-11T06:03:43.277106+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T06:03:43.277106+0000","last_undegraded":"2024-09-11T06:03:43.277106+0000","last_fullsized":"2024-09-11T06:03:43.277106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:55.262608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.125533+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T06:03:43.125533+0000","last_peered":"2024-09-11T06:03:43.125533+0000","last_clean":"2024-09-11T06:03:43.125533+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T06:03:43.125533+0000","last_undegraded":"2024-09-11T06:03:43.125533+0000","last_fullsized":"2024-09-11T06:03:43.125533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.124622+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T06:03:43.124622+0000","last_peered":"2024-09-11T06:03:43.124622+0000","last_clean":"2024-09-11T06:03:43.124622+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T06:03:43.124622+0000","last_undegraded":"2024-09-11T06:03:43.124622+0000","last_fullsized":"2024-09-11T06:03:43.124622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.569100+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T06:03:43.569100+0000","last_peered":"2024-09-11T06:03:43.569100+0000","last_clean":"2024-09-11T06:03:43.569100+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T06:03:43.569100+0000","last_undegraded":"2024-09-11T06:03:43.569100+0000","last_fullsized":"2024-09-11T06:03:43.569100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1143,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572040+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T06:03:43.572040+0000","last_peered":"2024-09-11T06:03:43.572040+0000","last_clean":"2024-09-11T06:03:43.572040+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T06:03:43.572040+0000","last_undegraded":"2024-09-11T06:03:43.572040+0000","last_fullsized":"2024-09-11T06:03:43.572040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1224940,"kb_used_data":5848,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":93146900,"statfs":{"total":96636764160,"available":95382425600,"internally_reserved":0,"allocated":5988352,"data_stored":2987851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1248328133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273640,"kb_used_data":5908,"kb_used_omap":1,"kb_used_meta":1267710,"kb_avail":93098200,"statfs":{"total":96636764160,"available":95332556800,"internally_reserved":0,"allocated":6049792,"data_stored":3578238,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1298135493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1468448,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":1462526,"kb_avail":92903392,"statfs":{"total":96636764160,"available":95133073408,"internally_reserved":0,"allocated":6045696,"data_stored":3578219,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1497627077},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:03:47.041 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:03:47.327 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:03:47.328 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":505,"fsid":"53ed04a9-aee2-4881-a1db-0ad67faa19f3","created":"2024-09-11T05:25:07.609634+0000","modified":"2024-09-11T06:03:43.111893+0000","last_up_change":"2024-09-11T05:25:17.333638+0000","last_in_change":"2024-09-11T05:25:08.529475+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:25:19.202066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:25:22.987864+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"730def7b-fac3-4e17-b5c0-bb811d132db0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6809","nonce":2274081021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6811","nonce":2274081021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6815","nonce":2274081021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":2274081021},{"type":"v1","addr":"172.21.15.164:6813","nonce":2274081021}]},"public_addr":"172.21.15.164:6809/2274081021","cluster_addr":"172.21.15.164:6811/2274081021","heartbeat_back_addr":"172.21.15.164:6815/2274081021","heartbeat_front_addr":"172.21.15.164:6813/2274081021","state":["exists","up"]},{"osd":1,"uuid":"1d736f34-58e5-4130-841c-7dbbf7ea593f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6801","nonce":22640096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6803","nonce":22640096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6807","nonce":22640096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22640096},{"type":"v1","addr":"172.21.15.164:6805","nonce":22640096}]},"public_addr":"172.21.15.164:6801/22640096","cluster_addr":"172.21.15.164:6803/22640096","heartbeat_back_addr":"172.21.15.164:6807/22640096","heartbeat_front_addr":"172.21.15.164:6805/22640096","state":["exists","up"]},{"osd":2,"uuid":"841ac8c5-41ad-496b-83b7-718ef75cc185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6817","nonce":3528912721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6819","nonce":3528912721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6823","nonce":3528912721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":3528912721},{"type":"v1","addr":"172.21.15.164:6821","nonce":3528912721}]},"public_addr":"172.21.15.164:6817/3528912721","cluster_addr":"172.21.15.164:6819/3528912721","heartbeat_back_addr":"172.21.15.164:6823/3528912721","heartbeat_front_addr":"172.21.15.164:6821/3528912721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.658323+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.789994+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:25:14.903487+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:0/5560995":"2024-09-11T07:03:34.850222+0000","172.21.15.164:0/2907192767":"2024-09-11T07:02:51.894211+0000","172.21.15.164:0/1139467851":"2024-09-11T07:02:04.209558+0000","172.21.15.164:0/764541488":"2024-09-11T07:01:23.115978+0000","172.21.15.164:0/256885925":"2024-09-11T07:00:46.767759+0000","172.21.15.164:0/903512131":"2024-09-11T06:59:16.021558+0000","172.21.15.164:0/4139504535":"2024-09-11T06:58:38.235200+0000","172.21.15.164:0/3196082015":"2024-09-11T06:53:45.086577+0000","172.21.15.164:0/2717801190":"2024-09-11T07:00:00.466807+0000","172.21.15.164:0/2605206718":"2024-09-11T06:48:50.133653+0000","172.21.15.164:0/1647510692":"2024-09-11T06:48:41.095662+0000","172.21.15.164:0/3306429969":"2024-09-11T06:27:55.873559+0000","172.21.15.164:0/2003594638":"2024-09-11T06:48:58.885551+0000","172.21.15.164:0/3240864504":"2024-09-11T06:48:14.021801+0000","172.21.15.164:0/2343033322":"2024-09-11T06:47:38.109179+0000","172.21.15.164:0/1224379417":"2024-09-11T06:52:35.245760+0000","172.21.15.164:0/1474210348":"2024-09-11T06:52:17.200651+0000","172.21.15.164:0/3510788545":"2024-09-11T06:29:39.698212+0000","172.21.15.164:0/11996720":"2024-09-11T06:47:55.986041+0000","172.21.15.164:0/2575340323":"2024-09-11T06:52:08.217135+0000","172.21.15.164:0/1980224239":"2024-09-11T06:47:29.018953+0000","172.21.15.164:0/1556885578":"2024-09-11T06:47:20.074067+0000","172.21.15.164:0/858424279":"2024-09-11T06:46:53.164822+0000","172.21.15.164:0/964781721":"2024-09-11T06:57:59.133940+0000","172.21.15.164:0/4022953677":"2024-09-11T06:48:32.094011+0000","172.21.15.164:0/1924259086":"2024-09-11T06:46:26.306090+0000","172.21.15.164:0/2103311186":"2024-09-11T06:49:07.982642+0000","172.21.15.164:0/2976272044":"2024-09-11T06:35:52.892538+0000","172.21.15.164:0/3995692704":"2024-09-11T06:35:03.449525+0000","172.21.15.164:0/666172794":"2024-09-11T06:30:30.583200+0000","172.21.15.164:0/4279403325":"2024-09-11T06:57:16.866474+0000","172.21.15.164:0/2311938236":"2024-09-11T06:50:47.162688+0000","172.21.15.164:0/2663565798":"2024-09-11T06:51:14.325002+0000","172.21.15.164:0/1361820815":"2024-09-11T06:48:05.266144+0000","172.21.15.164:0/3537024054":"2024-09-11T06:46:43.932191+0000","172.21.15.164:0/2892319958":"2024-09-11T06:31:26.343192+0000","172.21.15.164:0/1976951052":"2024-09-11T06:32:23.328362+0000","172.21.15.164:0/4246149166":"2024-09-11T06:49:53.227481+0000","172.21.15.164:0/3652299089":"2024-09-11T06:33:13.774703+0000","172.21.15.164:0/2505358391":"2024-09-11T06:49:44.160835+0000","172.21.15.164:0/1392037910":"2024-09-11T06:47:47.158563+0000","172.21.15.164:0/968347190":"2024-09-11T06:47:10.989208+0000","172.21.15.164:0/1919991810":"2024-09-11T06:34:08.457905+0000","172.21.15.164:0/3334255002":"2024-09-11T06:46:35.203148+0000","172.21.15.164:0/3574106734":"2024-09-11T06:49:17.137075+0000","172.21.15.164:0/2706945848":"2024-09-11T06:49:26.069779+0000","172.21.15.164:0/145235449":"2024-09-11T06:49:35.096420+0000","172.21.15.164:0/617437427":"2024-09-11T06:28:50.953453+0000","172.21.15.164:0/3638618156":"2024-09-11T06:53:20.452287+0000","172.21.15.164:0/2059840040":"2024-09-11T06:53:35.957012+0000","172.21.15.164:0/2785606337":"2024-09-11T06:50:02.368167+0000","172.21.15.164:0/2933699473":"2024-09-11T06:52:26.310547+0000","172.21.15.164:0/396153850":"2024-09-11T06:47:01.984169+0000","172.21.15.164:0/4229161879":"2024-09-11T06:50:10.993174+0000","172.21.15.164:0/1332956296":"2024-09-11T06:53:02.193963+0000","172.21.15.164:0/693625503":"2024-09-11T06:48:23.077710+0000","172.21.15.164:0/1747190896":"2024-09-11T06:50:38.085812+0000","172.21.15.164:0/375966110":"2024-09-11T06:51:05.188757+0000","172.21.15.164:0/33039408":"2024-09-11T06:51:23.399963+0000","172.21.15.164:0/1630711692":"2024-09-11T06:50:20.068105+0000","172.21.15.164:0/1732412907":"2024-09-11T06:51:32.194132+0000","172.21.15.164:0/3855961002":"2024-09-11T06:51:41.041432+0000","172.21.15.164:0/3704945490":"2024-09-11T06:51:50.311767+0000","172.21.15.164:0/2092080391":"2024-09-11T06:51:59.182011+0000","172.21.15.164:0/1682572508":"2024-09-11T06:52:44.043892+0000","172.21.15.164:0/3541846218":"2024-09-11T06:52:53.131777+0000","172.21.15.164:0/2934985110":"2024-09-11T06:50:56.123709+0000","172.21.15.164:0/3342925538":"2024-09-11T06:53:11.133733+0000","172.21.15.164:0/2985584809":"2024-09-11T06:50:28.966517+0000","172.21.15.164:0/2150523863":"2024-09-11T06:53:29.231575+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:03:48.346 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-11T06:03:48.346 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:03:48.507 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-11T06:03:48.508 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:03:48.508 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-11T06:03:48.523 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-11T06:03:48.813 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T06:03:48.832 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-11T06:03:48.832 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:03:48.982 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-11T06:03:48.982 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:03:48.982 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-11T06:03:48.998 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-11T06:03:49.278 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T06:03:49.297 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-11T06:03:49.297 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:03:49.440 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-11T06:03:49.440 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:03:49.440 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-11T06:03:49.456 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-11T06:03:49.743 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T06:03:49.761 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:03:50.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:03:50.048 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T06:03:50.065 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-09-11T06:03:50.000380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3967076,"kb_used_data":17700,"kb_used_omap":4,"kb_used_meta":3949307,"kb_avail":279148444,"statfs":{"total":289910292480,"available":285848006656,"internally_reserved":0,"allocated":18124800,"data_stored":10164274,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4044090703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2207,"num_read_kb":-2087,"num_write":-436,"num_write_kb":-328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.890335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572346+0000","last_change":"2024-09-11T05:25:26.390971+0000","last_active":"2024-09-11T06:03:43.572346+0000","last_peered":"2024-09-11T06:03:43.572346+0000","last_clean":"2024-09-11T06:03:43.572346+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T06:03:43.572346+0000","last_undegraded":"2024-09-11T06:03:43.572346+0000","last_fullsized":"2024-09-11T06:03:43.572346+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:16.883840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.571272+0000","last_change":"2024-09-11T05:25:26.390354+0000","last_active":"2024-09-11T06:03:43.571272+0000","last_peered":"2024-09-11T06:03:43.571272+0000","last_clean":"2024-09-11T06:03:43.571272+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T06:03:43.571272+0000","last_undegraded":"2024-09-11T06:03:43.571272+0000","last_fullsized":"2024-09-11T06:03:43.571272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:30.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572461+0000","last_change":"2024-09-11T05:25:26.390055+0000","last_active":"2024-09-11T06:03:43.572461+0000","last_peered":"2024-09-11T06:03:43.572461+0000","last_clean":"2024-09-11T06:03:43.572461+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T06:03:43.572461+0000","last_undegraded":"2024-09-11T06:03:43.572461+0000","last_fullsized":"2024-09-11T06:03:43.572461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:23.317121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.568538+0000","last_change":"2024-09-11T05:25:26.389660+0000","last_active":"2024-09-11T06:03:43.568538+0000","last_peered":"2024-09-11T06:03:43.568538+0000","last_clean":"2024-09-11T06:03:43.568538+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T06:03:43.568538+0000","last_undegraded":"2024-09-11T06:03:43.568538+0000","last_fullsized":"2024-09-11T06:03:43.568538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:41:19.646793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1018,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.277162+0000","last_change":"2024-09-11T05:25:26.382913+0000","last_active":"2024-09-11T06:03:43.277162+0000","last_peered":"2024-09-11T06:03:43.277162+0000","last_clean":"2024-09-11T06:03:43.277162+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T06:03:43.277162+0000","last_undegraded":"2024-09-11T06:03:43.277162+0000","last_fullsized":"2024-09-11T06:03:43.277162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.125533+0000","last_change":"2024-09-11T05:25:26.878948+0000","last_active":"2024-09-11T06:03:43.125533+0000","last_peered":"2024-09-11T06:03:43.125533+0000","last_clean":"2024-09-11T06:03:43.125533+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T06:03:43.125533+0000","last_undegraded":"2024-09-11T06:03:43.125533+0000","last_fullsized":"2024-09-11T06:03:43.125533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:36:37.632678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.124622+0000","last_change":"2024-09-11T05:25:26.878938+0000","last_active":"2024-09-11T06:03:43.124622+0000","last_peered":"2024-09-11T06:03:43.124622+0000","last_clean":"2024-09-11T06:03:43.124622+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T06:03:43.124622+0000","last_undegraded":"2024-09-11T06:03:43.124622+0000","last_fullsized":"2024-09-11T06:03:43.124622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.263282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.569100+0000","last_change":"2024-09-11T05:25:26.389715+0000","last_active":"2024-09-11T06:03:43.569100+0000","last_peered":"2024-09-11T06:03:43.569100+0000","last_clean":"2024-09-11T06:03:43.569100+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T06:03:43.569100+0000","last_undegraded":"2024-09-11T06:03:43.569100+0000","last_fullsized":"2024-09-11T06:03:43.569100+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:23.357786+0000","last_clean_scrub_stamp":"2024-09-11T05:25:23.357786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:07:18.269292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":1143,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:43.572040+0000","last_change":"2024-09-11T05:25:20.768313+0000","last_active":"2024-09-11T06:03:43.572040+0000","last_peered":"2024-09-11T06:03:43.572040+0000","last_clean":"2024-09-11T06:03:43.572040+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T06:03:43.572040+0000","last_undegraded":"2024-09-11T06:03:43.572040+0000","last_fullsized":"2024-09-11T06:03:43.572040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:25:19.337744+0000","last_clean_scrub_stamp":"2024-09-11T05:25:19.337744+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:11.907545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1224956,"kb_used_data":5860,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":93146884,"statfs":{"total":96636764160,"available":95382409216,"internally_reserved":0,"allocated":6000640,"data_stored":2994500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1248328133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273652,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1267710,"kb_avail":93098188,"statfs":{"total":96636764160,"available":95332544512,"internally_reserved":0,"allocated":6062080,"data_stored":3584887,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1298135493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1468468,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1462526,"kb_avail":92903372,"statfs":{"total":96636764160,"available":95133052928,"internally_reserved":0,"allocated":6062080,"data_stored":3584887,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1497627077},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:03:50.068 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.068 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.068 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T05:25:23.357786+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T05:25:19.337744+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=25, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=3, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:03:50.069 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T06:04:10.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:04:10.381 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-11T06:04:10.382 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-11T06:04:10.400 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-09-11T06:04:10.005276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3967060,"kb_used_data":17700,"kb_used_omap":4,"kb_used_meta":3949307,"kb_avail":279148460,"statfs":{"total":289910292480,"available":285848023040,"internally_reserved":0,"allocated":18124800,"data_stored":10164274,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4044090703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:53.143695+0000","last_change":"2024-09-11T06:03:53.143750+0000","last_active":"2024-09-11T06:03:53.143695+0000","last_peered":"2024-09-11T06:03:53.143695+0000","last_clean":"2024-09-11T06:03:53.143695+0000","last_became_active":"2024-09-11T05:25:24.385224+0000","last_became_peered":"2024-09-11T05:25:24.385224+0000","last_unstale":"2024-09-11T06:03:53.143695+0000","last_undegraded":"2024-09-11T06:03:53.143695+0000","last_fullsized":"2024-09-11T06:03:53.143695+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:53.143503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:53.143503+0000","last_clean_scrub_stamp":"2024-09-11T06:03:53.143503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:04:25.501310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020344550000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:54.126044+0000","last_change":"2024-09-11T06:03:54.126100+0000","last_active":"2024-09-11T06:03:54.126044+0000","last_peered":"2024-09-11T06:03:54.126044+0000","last_clean":"2024-09-11T06:03:54.126044+0000","last_became_active":"2024-09-11T05:25:24.385522+0000","last_became_peered":"2024-09-11T05:25:24.385522+0000","last_unstale":"2024-09-11T06:03:54.126044+0000","last_undegraded":"2024-09-11T06:03:54.126044+0000","last_fullsized":"2024-09-11T06:03:54.126044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:54.125829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:54.125829+0000","last_clean_scrub_stamp":"2024-09-11T06:03:54.125829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:03:40.471297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001710196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:51.139381+0000","last_change":"2024-09-11T06:03:51.139445+0000","last_active":"2024-09-11T06:03:51.139381+0000","last_peered":"2024-09-11T06:03:51.139381+0000","last_clean":"2024-09-11T06:03:51.139381+0000","last_became_active":"2024-09-11T05:25:24.379942+0000","last_became_peered":"2024-09-11T05:25:24.379942+0000","last_unstale":"2024-09-11T06:03:51.139381+0000","last_undegraded":"2024-09-11T06:03:51.139381+0000","last_fullsized":"2024-09-11T06:03:51.139381+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:51.139114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:51.139114+0000","last_clean_scrub_stamp":"2024-09-11T06:03:51.139114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:43:11.361063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001698211,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:55.167225+0000","last_change":"2024-09-11T06:03:55.167284+0000","last_active":"2024-09-11T06:03:55.167225+0000","last_peered":"2024-09-11T06:03:55.167225+0000","last_clean":"2024-09-11T06:03:55.167225+0000","last_became_active":"2024-09-11T05:25:24.386120+0000","last_became_peered":"2024-09-11T05:25:24.386120+0000","last_unstale":"2024-09-11T06:03:55.167225+0000","last_undegraded":"2024-09-11T06:03:55.167225+0000","last_fullsized":"2024-09-11T06:03:55.167225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:55.167021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:55.167021+0000","last_clean_scrub_stamp":"2024-09-11T06:03:55.167021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:09:01.707476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018654419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"505'3","reported_seq":1027,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:49.110214+0000","last_change":"2024-09-11T06:03:49.110267+0000","last_active":"2024-09-11T06:03:49.110214+0000","last_peered":"2024-09-11T06:03:49.110214+0000","last_clean":"2024-09-11T06:03:49.110214+0000","last_became_active":"2024-09-11T05:25:24.382811+0000","last_became_peered":"2024-09-11T05:25:24.382811+0000","last_unstale":"2024-09-11T06:03:49.110214+0000","last_undegraded":"2024-09-11T06:03:49.110214+0000","last_fullsized":"2024-09-11T06:03:49.110214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'3","last_scrub_stamp":"2024-09-11T06:03:49.110123+0000","last_deep_scrub":"505'3","last_deep_scrub_stamp":"2024-09-11T06:03:49.110123+0000","last_clean_scrub_stamp":"2024-09-11T06:03:49.110123+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:12:23.110849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00078454700000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1022,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:50.728365+0000","last_change":"2024-09-11T06:03:50.728419+0000","last_active":"2024-09-11T06:03:50.728365+0000","last_peered":"2024-09-11T06:03:50.728365+0000","last_clean":"2024-09-11T06:03:50.728365+0000","last_became_active":"2024-09-11T05:25:24.383066+0000","last_became_peered":"2024-09-11T05:25:24.383066+0000","last_unstale":"2024-09-11T06:03:50.728365+0000","last_undegraded":"2024-09-11T06:03:50.728365+0000","last_fullsized":"2024-09-11T06:03:50.728365+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:50.728179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:50.728179+0000","last_clean_scrub_stamp":"2024-09-11T06:03:50.728179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:23:22.382447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00033374499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1023,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:49.753819+0000","last_change":"2024-09-11T06:03:49.753846+0000","last_active":"2024-09-11T06:03:49.753819+0000","last_peered":"2024-09-11T06:03:49.753819+0000","last_clean":"2024-09-11T06:03:49.753819+0000","last_became_active":"2024-09-11T05:25:24.381172+0000","last_became_peered":"2024-09-11T05:25:24.381172+0000","last_unstale":"2024-09-11T06:03:49.753819+0000","last_undegraded":"2024-09-11T06:03:49.753819+0000","last_fullsized":"2024-09-11T06:03:49.753819+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:03:49.753720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:03:49.753720+0000","last_clean_scrub_stamp":"2024-09-11T06:03:49.753720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:50:02.225057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033261800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"505'2","reported_seq":1025,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:52.120438+0000","last_change":"2024-09-11T06:03:52.120584+0000","last_active":"2024-09-11T06:03:52.120438+0000","last_peered":"2024-09-11T06:03:52.120438+0000","last_clean":"2024-09-11T06:03:52.120438+0000","last_became_active":"2024-09-11T05:25:24.379819+0000","last_became_peered":"2024-09-11T05:25:24.379819+0000","last_unstale":"2024-09-11T06:03:52.120438+0000","last_undegraded":"2024-09-11T06:03:52.120438+0000","last_fullsized":"2024-09-11T06:03:52.120438+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'2","last_scrub_stamp":"2024-09-11T06:03:52.120203+0000","last_deep_scrub":"505'2","last_deep_scrub_stamp":"2024-09-11T06:03:52.120203+0000","last_clean_scrub_stamp":"2024-09-11T06:03:52.120203+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:56.422799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001836227,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"505'99","reported_seq":1154,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-11T06:03:50.098527+0000","last_change":"2024-09-11T06:03:50.098559+0000","last_active":"2024-09-11T06:03:50.098527+0000","last_peered":"2024-09-11T06:03:50.098527+0000","last_clean":"2024-09-11T06:03:50.098527+0000","last_became_active":"2024-09-11T05:25:20.765973+0000","last_became_peered":"2024-09-11T05:25:20.765973+0000","last_unstale":"2024-09-11T06:03:50.098527+0000","last_undegraded":"2024-09-11T06:03:50.098527+0000","last_fullsized":"2024-09-11T06:03:50.098527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'99","last_scrub_stamp":"2024-09-11T06:03:50.098465+0000","last_deep_scrub":"505'99","last_deep_scrub_stamp":"2024-09-11T06:03:50.098465+0000","last_clean_scrub_stamp":"2024-09-11T06:03:50.098465+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:35.436872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12514,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1224948,"kb_used_data":5860,"kb_used_omap":1,"kb_used_meta":1219070,"kb_avail":93146892,"statfs":{"total":96636764160,"available":95382417408,"internally_reserved":0,"allocated":6000640,"data_stored":2994500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1248328133},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273648,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1267710,"kb_avail":93098192,"statfs":{"total":96636764160,"available":95332548608,"internally_reserved":0,"allocated":6062080,"data_stored":3584887,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1298135493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1468464,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1462526,"kb_avail":92903376,"statfs":{"total":96636764160,"available":95133057024,"internally_reserved":0,"allocated":6062080,"data_stored":3584887,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1497627077},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6257,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:10.401 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-11T06:04:10.724 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T06:04:10.724 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T06:04:10.724 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-11T06:04:10.724 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:15.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-11T06:04:16.826 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-11T06:04:16.826 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-11T06:04:16.826 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:20.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-09-11T06:04:20.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-11T06:04:22.928 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-11T06:04:22.928 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-11T06:04:22.928 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:25.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-09-11T06:04:25.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-09-11T06:04:29.030 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-11T06:04:29.031 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T06:04:29.032 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-11T06:04:29.032 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:29.060 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-11T06:04:29.060 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T06:04:29.060 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-11T06:04:29.060 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:29.085 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-11T06:04:29.085 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T06:04:29.086 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:04:29.180 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T06:04:29.180 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-11T06:04:29.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T06:04:29.424 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-11T06:04:29.624 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T06:04:29.624 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-11T06:04:29.856 INFO:tasks.ceph:Archiving mon data... 2024-09-11T06:04:29.857 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/data/cluster2.mon.a.tgz 2024-09-11T06:04:29.858 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-09-11T06:04:29.862 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.VMcgruhJum 2024-09-11T06:04:29.863 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.VMcgruhJum 2024-09-11T06:04:30.035 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.VMcgruhJum 2024-09-11T06:04:30.141 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.VMcgruhJum is 466KB 2024-09-11T06:04:30.202 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.VMcgruhJum 2024-09-11T06:04:30.207 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T06:04:30.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:04:30.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:04:30.261 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T06:04:30.273 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T06:04:30.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:04:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:30.582 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T06:04:30.599 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-09-11T06:04:30.028196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":438016,"kb_used_data":11328,"kb_used_omap":4,"kb_used_meta":426619,"kb_avail":282677504,"statfs":{"total":289910292480,"available":289461764096,"internally_reserved":0,"allocated":11599872,"data_stored":7311049,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":436858191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838864+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T06:03:41.838864+0000","last_peered":"2024-09-11T06:03:41.838864+0000","last_clean":"2024-09-11T06:03:41.838864+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T06:03:41.838864+0000","last_undegraded":"2024-09-11T06:03:41.838864+0000","last_fullsized":"2024-09-11T06:03:41.838864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.842598+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T06:03:41.842598+0000","last_peered":"2024-09-11T06:03:41.842598+0000","last_clean":"2024-09-11T06:03:41.842598+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T06:03:41.842598+0000","last_undegraded":"2024-09-11T06:03:41.842598+0000","last_fullsized":"2024-09-11T06:03:41.842598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.841633+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T06:03:41.841633+0000","last_peered":"2024-09-11T06:03:41.841633+0000","last_clean":"2024-09-11T06:03:41.841633+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T06:03:41.841633+0000","last_undegraded":"2024-09-11T06:03:41.841633+0000","last_fullsized":"2024-09-11T06:03:41.841633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.830869+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T06:03:41.830869+0000","last_peered":"2024-09-11T06:03:41.830869+0000","last_clean":"2024-09-11T06:03:41.830869+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T06:03:41.830869+0000","last_undegraded":"2024-09-11T06:03:41.830869+0000","last_fullsized":"2024-09-11T06:03:41.830869+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.879074+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T06:03:42.879074+0000","last_peered":"2024-09-11T06:03:42.879074+0000","last_clean":"2024-09-11T06:03:42.879074+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T06:03:42.879074+0000","last_undegraded":"2024-09-11T06:03:42.879074+0000","last_fullsized":"2024-09-11T06:03:42.879074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.876341+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T06:03:42.876341+0000","last_peered":"2024-09-11T06:03:42.876341+0000","last_clean":"2024-09-11T06:03:42.876341+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T06:03:42.876341+0000","last_undegraded":"2024-09-11T06:03:42.876341+0000","last_fullsized":"2024-09-11T06:03:42.876341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.874555+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T06:03:42.874555+0000","last_peered":"2024-09-11T06:03:42.874555+0000","last_clean":"2024-09-11T06:03:42.874555+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T06:03:42.874555+0000","last_undegraded":"2024-09-11T06:03:42.874555+0000","last_fullsized":"2024-09-11T06:03:42.874555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.833104+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T06:03:41.833104+0000","last_peered":"2024-09-11T06:03:41.833104+0000","last_clean":"2024-09-11T06:03:41.833104+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T06:03:41.833104+0000","last_undegraded":"2024-09-11T06:03:41.833104+0000","last_fullsized":"2024-09-11T06:03:41.833104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838414+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T06:03:41.838414+0000","last_peered":"2024-09-11T06:03:41.838414+0000","last_clean":"2024-09-11T06:03:41.838414+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T06:03:41.838414+0000","last_undegraded":"2024-09-11T06:03:41.838414+0000","last_fullsized":"2024-09-11T06:03:41.838414+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125100,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246740,"statfs":{"total":96636764160,"available":96508661760,"internally_reserved":0,"allocated":3825664,"data_stored":2043425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706146,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147432,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94224408,"statfs":{"total":96636764160,"available":96485793792,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706146,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165484,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206356,"statfs":{"total":96636764160,"available":96467308544,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:30.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:04:30.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-09-11T06:04:30.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:30.898 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T06:04:30.916 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-09-11T06:04:30.028196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":438016,"kb_used_data":11328,"kb_used_omap":4,"kb_used_meta":426619,"kb_avail":282677504,"statfs":{"total":289910292480,"available":289461764096,"internally_reserved":0,"allocated":11599872,"data_stored":7311049,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":436858191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838864+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T06:03:41.838864+0000","last_peered":"2024-09-11T06:03:41.838864+0000","last_clean":"2024-09-11T06:03:41.838864+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T06:03:41.838864+0000","last_undegraded":"2024-09-11T06:03:41.838864+0000","last_fullsized":"2024-09-11T06:03:41.838864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.842598+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T06:03:41.842598+0000","last_peered":"2024-09-11T06:03:41.842598+0000","last_clean":"2024-09-11T06:03:41.842598+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T06:03:41.842598+0000","last_undegraded":"2024-09-11T06:03:41.842598+0000","last_fullsized":"2024-09-11T06:03:41.842598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.841633+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T06:03:41.841633+0000","last_peered":"2024-09-11T06:03:41.841633+0000","last_clean":"2024-09-11T06:03:41.841633+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T06:03:41.841633+0000","last_undegraded":"2024-09-11T06:03:41.841633+0000","last_fullsized":"2024-09-11T06:03:41.841633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.830869+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T06:03:41.830869+0000","last_peered":"2024-09-11T06:03:41.830869+0000","last_clean":"2024-09-11T06:03:41.830869+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T06:03:41.830869+0000","last_undegraded":"2024-09-11T06:03:41.830869+0000","last_fullsized":"2024-09-11T06:03:41.830869+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.879074+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T06:03:42.879074+0000","last_peered":"2024-09-11T06:03:42.879074+0000","last_clean":"2024-09-11T06:03:42.879074+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T06:03:42.879074+0000","last_undegraded":"2024-09-11T06:03:42.879074+0000","last_fullsized":"2024-09-11T06:03:42.879074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.876341+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T06:03:42.876341+0000","last_peered":"2024-09-11T06:03:42.876341+0000","last_clean":"2024-09-11T06:03:42.876341+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T06:03:42.876341+0000","last_undegraded":"2024-09-11T06:03:42.876341+0000","last_fullsized":"2024-09-11T06:03:42.876341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.874555+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T06:03:42.874555+0000","last_peered":"2024-09-11T06:03:42.874555+0000","last_clean":"2024-09-11T06:03:42.874555+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T06:03:42.874555+0000","last_undegraded":"2024-09-11T06:03:42.874555+0000","last_fullsized":"2024-09-11T06:03:42.874555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.833104+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T06:03:41.833104+0000","last_peered":"2024-09-11T06:03:41.833104+0000","last_clean":"2024-09-11T06:03:41.833104+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T06:03:41.833104+0000","last_undegraded":"2024-09-11T06:03:41.833104+0000","last_fullsized":"2024-09-11T06:03:41.833104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838414+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T06:03:41.838414+0000","last_peered":"2024-09-11T06:03:41.838414+0000","last_clean":"2024-09-11T06:03:41.838414+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T06:03:41.838414+0000","last_undegraded":"2024-09-11T06:03:41.838414+0000","last_fullsized":"2024-09-11T06:03:41.838414+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125100,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246740,"statfs":{"total":96636764160,"available":96508661760,"internally_reserved":0,"allocated":3825664,"data_stored":2043425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706146,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147432,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94224408,"statfs":{"total":96636764160,"available":96485793792,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706146,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165484,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206356,"statfs":{"total":96636764160,"available":96467308544,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:30.917 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T06:04:30.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:04:31.211 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T06:04:31.230 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-09-11T06:04:30.028196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":438016,"kb_used_data":11328,"kb_used_omap":4,"kb_used_meta":426619,"kb_avail":282677504,"statfs":{"total":289910292480,"available":289461764096,"internally_reserved":0,"allocated":11599872,"data_stored":7311049,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":436858191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002883"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838864+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T06:03:41.838864+0000","last_peered":"2024-09-11T06:03:41.838864+0000","last_clean":"2024-09-11T06:03:41.838864+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T06:03:41.838864+0000","last_undegraded":"2024-09-11T06:03:41.838864+0000","last_fullsized":"2024-09-11T06:03:41.838864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.842598+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T06:03:41.842598+0000","last_peered":"2024-09-11T06:03:41.842598+0000","last_clean":"2024-09-11T06:03:41.842598+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T06:03:41.842598+0000","last_undegraded":"2024-09-11T06:03:41.842598+0000","last_fullsized":"2024-09-11T06:03:41.842598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.841633+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T06:03:41.841633+0000","last_peered":"2024-09-11T06:03:41.841633+0000","last_clean":"2024-09-11T06:03:41.841633+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T06:03:41.841633+0000","last_undegraded":"2024-09-11T06:03:41.841633+0000","last_fullsized":"2024-09-11T06:03:41.841633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.830869+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T06:03:41.830869+0000","last_peered":"2024-09-11T06:03:41.830869+0000","last_clean":"2024-09-11T06:03:41.830869+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T06:03:41.830869+0000","last_undegraded":"2024-09-11T06:03:41.830869+0000","last_fullsized":"2024-09-11T06:03:41.830869+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.879074+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T06:03:42.879074+0000","last_peered":"2024-09-11T06:03:42.879074+0000","last_clean":"2024-09-11T06:03:42.879074+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T06:03:42.879074+0000","last_undegraded":"2024-09-11T06:03:42.879074+0000","last_fullsized":"2024-09-11T06:03:42.879074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.876341+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T06:03:42.876341+0000","last_peered":"2024-09-11T06:03:42.876341+0000","last_clean":"2024-09-11T06:03:42.876341+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T06:03:42.876341+0000","last_undegraded":"2024-09-11T06:03:42.876341+0000","last_fullsized":"2024-09-11T06:03:42.876341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.874555+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T06:03:42.874555+0000","last_peered":"2024-09-11T06:03:42.874555+0000","last_clean":"2024-09-11T06:03:42.874555+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T06:03:42.874555+0000","last_undegraded":"2024-09-11T06:03:42.874555+0000","last_fullsized":"2024-09-11T06:03:42.874555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.833104+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T06:03:41.833104+0000","last_peered":"2024-09-11T06:03:41.833104+0000","last_clean":"2024-09-11T06:03:41.833104+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T06:03:41.833104+0000","last_undegraded":"2024-09-11T06:03:41.833104+0000","last_fullsized":"2024-09-11T06:03:41.833104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838414+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T06:03:41.838414+0000","last_peered":"2024-09-11T06:03:41.838414+0000","last_clean":"2024-09-11T06:03:41.838414+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T06:03:41.838414+0000","last_undegraded":"2024-09-11T06:03:41.838414+0000","last_fullsized":"2024-09-11T06:03:41.838414+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125100,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246740,"statfs":{"total":96636764160,"available":96508661760,"internally_reserved":0,"allocated":3825664,"data_stored":2043425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706146,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147432,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94224408,"statfs":{"total":96636764160,"available":96485793792,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706146,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165484,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206356,"statfs":{"total":96636764160,"available":96467308544,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:31.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:04:31.533 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:31.533 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":465,"fsid":"aeb205b2-dc94-4e51-98a0-6978e8bdb941","created":"2024-09-11T05:24:28.315967+0000","modified":"2024-09-11T06:03:41.813152+0000","last_up_change":"2024-09-11T05:24:38.228180+0000","last_in_change":"2024-09-11T05:24:29.239161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:24:39.240696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:24:43.666844+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c33b7ad-c171-474c-8a11-23bdde4e7282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6801","nonce":4088995954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6803","nonce":4088995954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6807","nonce":4088995954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":4088995954},{"type":"v1","addr":"172.21.15.67:6805","nonce":4088995954}]},"public_addr":"172.21.15.67:6801/4088995954","cluster_addr":"172.21.15.67:6803/4088995954","heartbeat_back_addr":"172.21.15.67:6807/4088995954","heartbeat_front_addr":"172.21.15.67:6805/4088995954","state":["exists","up"]},{"osd":1,"uuid":"f39a9aee-6278-415d-bf58-d41b43525a6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6809","nonce":2056339409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6811","nonce":2056339409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6815","nonce":2056339409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2056339409},{"type":"v1","addr":"172.21.15.67:6813","nonce":2056339409}]},"public_addr":"172.21.15.67:6809/2056339409","cluster_addr":"172.21.15.67:6811/2056339409","heartbeat_back_addr":"172.21.15.67:6815/2056339409","heartbeat_front_addr":"172.21.15.67:6813/2056339409","state":["exists","up"]},{"osd":2,"uuid":"686e2789-7c26-4e0e-bec8-516001c9645e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6817","nonce":4074589272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6819","nonce":4074589272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6823","nonce":4074589272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4074589272},{"type":"v1","addr":"172.21.15.67:6821","nonce":4074589272}]},"public_addr":"172.21.15.67:6817/4074589272","cluster_addr":"172.21.15.67:6819/4074589272","heartbeat_back_addr":"172.21.15.67:6823/4074589272","heartbeat_front_addr":"172.21.15.67:6821/4074589272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.591349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:35.668801+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:24:36.227070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:04:32.553 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-11T06:04:32.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:04:32.718 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-11T06:04:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:04:32.718 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-11T06:04:32.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-11T06:04:33.023 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T06:04:33.041 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-11T06:04:33.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:04:33.180 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-11T06:04:33.180 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:04:33.180 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-11T06:04:33.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-11T06:04:33.481 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T06:04:33.501 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-11T06:04:33.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:04:33.646 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-11T06:04:33.647 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:04:33.647 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-11T06:04:33.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-11T06:04:33.957 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T06:04:33.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:04:34.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:34.266 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T06:04:34.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2024-09-11T06:04:34.029174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":438016,"kb_used_data":11328,"kb_used_omap":4,"kb_used_meta":426619,"kb_avail":282677504,"statfs":{"total":289910292480,"available":289461764096,"internally_reserved":0,"allocated":11599872,"data_stored":7311049,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":436858191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838864+0000","last_change":"2024-09-11T05:24:47.391805+0000","last_active":"2024-09-11T06:03:41.838864+0000","last_peered":"2024-09-11T06:03:41.838864+0000","last_clean":"2024-09-11T06:03:41.838864+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T06:03:41.838864+0000","last_undegraded":"2024-09-11T06:03:41.838864+0000","last_fullsized":"2024-09-11T06:03:41.838864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:22:36.204105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":934,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.842598+0000","last_change":"2024-09-11T05:24:47.392518+0000","last_active":"2024-09-11T06:03:41.842598+0000","last_peered":"2024-09-11T06:03:41.842598+0000","last_clean":"2024-09-11T06:03:41.842598+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T06:03:41.842598+0000","last_undegraded":"2024-09-11T06:03:41.842598+0000","last_fullsized":"2024-09-11T06:03:41.842598+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:26.249647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.841633+0000","last_change":"2024-09-11T05:24:47.391169+0000","last_active":"2024-09-11T06:03:41.841633+0000","last_peered":"2024-09-11T06:03:41.841633+0000","last_clean":"2024-09-11T06:03:41.841633+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T06:03:41.841633+0000","last_undegraded":"2024-09-11T06:03:41.841633+0000","last_fullsized":"2024-09-11T06:03:41.841633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:26:09.555022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":935,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.830869+0000","last_change":"2024-09-11T05:24:47.391956+0000","last_active":"2024-09-11T06:03:41.830869+0000","last_peered":"2024-09-11T06:03:41.830869+0000","last_clean":"2024-09-11T06:03:41.830869+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T06:03:41.830869+0000","last_undegraded":"2024-09-11T06:03:41.830869+0000","last_fullsized":"2024-09-11T06:03:41.830869+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:11.972109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":938,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.879074+0000","last_change":"2024-09-11T05:24:47.385463+0000","last_active":"2024-09-11T06:03:42.879074+0000","last_peered":"2024-09-11T06:03:42.879074+0000","last_clean":"2024-09-11T06:03:42.879074+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T06:03:42.879074+0000","last_undegraded":"2024-09-11T06:03:42.879074+0000","last_fullsized":"2024-09-11T06:03:42.879074+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:09.843120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.876341+0000","last_change":"2024-09-11T05:24:47.377777+0000","last_active":"2024-09-11T06:03:42.876341+0000","last_peered":"2024-09-11T06:03:42.876341+0000","last_clean":"2024-09-11T06:03:42.876341+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T06:03:42.876341+0000","last_undegraded":"2024-09-11T06:03:42.876341+0000","last_fullsized":"2024-09-11T06:03:42.876341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:04:19.695784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:42.874555+0000","last_change":"2024-09-11T05:24:47.377540+0000","last_active":"2024-09-11T06:03:42.874555+0000","last_peered":"2024-09-11T06:03:42.874555+0000","last_clean":"2024-09-11T06:03:42.874555+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T06:03:42.874555+0000","last_undegraded":"2024-09-11T06:03:42.874555+0000","last_fullsized":"2024-09-11T06:03:42.874555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:40.205542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.833104+0000","last_change":"2024-09-11T05:24:47.390942+0000","last_active":"2024-09-11T06:03:41.833104+0000","last_peered":"2024-09-11T06:03:41.833104+0000","last_clean":"2024-09-11T06:03:41.833104+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T06:03:41.833104+0000","last_undegraded":"2024-09-11T06:03:41.833104+0000","last_fullsized":"2024-09-11T06:03:41.833104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:44.340941+0000","last_clean_scrub_stamp":"2024-09-11T05:24:44.340941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:02:22.968890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1063,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:03:41.838414+0000","last_change":"2024-09-11T05:24:40.655836+0000","last_active":"2024-09-11T06:03:41.838414+0000","last_peered":"2024-09-11T06:03:41.838414+0000","last_clean":"2024-09-11T06:03:41.838414+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T06:03:41.838414+0000","last_undegraded":"2024-09-11T06:03:41.838414+0000","last_fullsized":"2024-09-11T06:03:41.838414+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:24:39.241302+0000","last_clean_scrub_stamp":"2024-09-11T05:24:39.241302+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:51:23.934090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125100,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246740,"statfs":{"total":96636764160,"available":96508661760,"internally_reserved":0,"allocated":3825664,"data_stored":2043425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706146,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147432,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94224408,"statfs":{"total":96636764160,"available":96485793792,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706146,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165484,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206356,"statfs":{"total":96636764160,"available":96467308544,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:34.284 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.284 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.284 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.284 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T05:24:44.340941+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T05:24:39.241302+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=24, tm_sec=39, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=4, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:04:34.285 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T06:04:54.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:04:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-11T06:04:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-11T06:04:54.625 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1670,"stamp":"2024-09-11T06:04:54.034009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":438000,"kb_used_data":11328,"kb_used_omap":4,"kb_used_meta":426619,"kb_avail":282677520,"statfs":{"total":289910292480,"available":289461780480,"internally_reserved":0,"allocated":11599872,"data_stored":7311049,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":436858191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":943,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:38.639322+0000","last_change":"2024-09-11T06:04:38.639377+0000","last_active":"2024-09-11T06:04:38.639322+0000","last_peered":"2024-09-11T06:04:38.639322+0000","last_clean":"2024-09-11T06:04:38.639322+0000","last_became_active":"2024-09-11T05:24:45.371213+0000","last_became_peered":"2024-09-11T05:24:45.371213+0000","last_unstale":"2024-09-11T06:04:38.639322+0000","last_undegraded":"2024-09-11T06:04:38.639322+0000","last_fullsized":"2024-09-11T06:04:38.639322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:38.639095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:38.639095+0000","last_clean_scrub_stamp":"2024-09-11T06:04:38.639095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:53:12.033025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001642674,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":943,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:36.639103+0000","last_change":"2024-09-11T06:04:36.639212+0000","last_active":"2024-09-11T06:04:36.639103+0000","last_peered":"2024-09-11T06:04:36.639103+0000","last_clean":"2024-09-11T06:04:36.639103+0000","last_became_active":"2024-09-11T05:24:45.370064+0000","last_became_peered":"2024-09-11T05:24:45.370064+0000","last_unstale":"2024-09-11T06:04:36.639103+0000","last_undegraded":"2024-09-11T06:04:36.639103+0000","last_fullsized":"2024-09-11T06:04:36.639103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:36.638873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:36.638873+0000","last_clean_scrub_stamp":"2024-09-11T06:04:36.638873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:25:45.236852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019185820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:37.615410+0000","last_change":"2024-09-11T06:04:37.615512+0000","last_active":"2024-09-11T06:04:37.615410+0000","last_peered":"2024-09-11T06:04:37.615410+0000","last_clean":"2024-09-11T06:04:37.615410+0000","last_became_active":"2024-09-11T05:24:45.370319+0000","last_became_peered":"2024-09-11T05:24:45.370319+0000","last_unstale":"2024-09-11T06:04:37.615410+0000","last_undegraded":"2024-09-11T06:04:37.615410+0000","last_fullsized":"2024-09-11T06:04:37.615410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:37.615159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:37.615159+0000","last_clean_scrub_stamp":"2024-09-11T06:04:37.615159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:00:33.780257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001145912,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":944,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:35.605389+0000","last_change":"2024-09-11T06:04:35.605448+0000","last_active":"2024-09-11T06:04:35.605389+0000","last_peered":"2024-09-11T06:04:35.605389+0000","last_clean":"2024-09-11T06:04:35.605389+0000","last_became_active":"2024-09-11T05:24:45.370276+0000","last_became_peered":"2024-09-11T05:24:45.370276+0000","last_unstale":"2024-09-11T06:04:35.605389+0000","last_undegraded":"2024-09-11T06:04:35.605389+0000","last_fullsized":"2024-09-11T06:04:35.605389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:35.605193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:35.605193+0000","last_clean_scrub_stamp":"2024-09-11T06:04:35.605193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:38.874603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001494146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"465'3","reported_seq":948,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:33.971746+0000","last_change":"2024-09-11T06:04:33.971770+0000","last_active":"2024-09-11T06:04:33.971746+0000","last_peered":"2024-09-11T06:04:33.971746+0000","last_clean":"2024-09-11T06:04:33.971746+0000","last_became_active":"2024-09-11T05:24:45.367992+0000","last_became_peered":"2024-09-11T05:24:45.367992+0000","last_unstale":"2024-09-11T06:04:33.971746+0000","last_undegraded":"2024-09-11T06:04:33.971746+0000","last_fullsized":"2024-09-11T06:04:33.971746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'3","last_scrub_stamp":"2024-09-11T06:04:33.971679+0000","last_deep_scrub":"465'3","last_deep_scrub_stamp":"2024-09-11T06:04:33.971679+0000","last_clean_scrub_stamp":"2024-09-11T06:04:33.971679+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:11:55.936054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023514550000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":945,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:34.175668+0000","last_change":"2024-09-11T06:04:34.175762+0000","last_active":"2024-09-11T06:04:34.175668+0000","last_peered":"2024-09-11T06:04:34.175668+0000","last_clean":"2024-09-11T06:04:34.175668+0000","last_became_active":"2024-09-11T05:24:45.367165+0000","last_became_peered":"2024-09-11T05:24:45.367165+0000","last_unstale":"2024-09-11T06:04:34.175668+0000","last_undegraded":"2024-09-11T06:04:34.175668+0000","last_fullsized":"2024-09-11T06:04:34.175668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:34.175455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:34.175455+0000","last_clean_scrub_stamp":"2024-09-11T06:04:34.175455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:15:32.286799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00089002299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":945,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:35.202513+0000","last_change":"2024-09-11T06:04:35.202642+0000","last_active":"2024-09-11T06:04:35.202513+0000","last_peered":"2024-09-11T06:04:35.202513+0000","last_clean":"2024-09-11T06:04:35.202513+0000","last_became_active":"2024-09-11T05:24:45.367234+0000","last_became_peered":"2024-09-11T05:24:45.367234+0000","last_unstale":"2024-09-11T06:04:35.202513+0000","last_undegraded":"2024-09-11T06:04:35.202513+0000","last_fullsized":"2024-09-11T06:04:35.202513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:04:35.202261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:04:35.202261+0000","last_clean_scrub_stamp":"2024-09-11T06:04:35.202261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:16.454900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074865000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"465'2","reported_seq":946,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:34.567181+0000","last_change":"2024-09-11T06:04:34.567333+0000","last_active":"2024-09-11T06:04:34.567181+0000","last_peered":"2024-09-11T06:04:34.567181+0000","last_clean":"2024-09-11T06:04:34.567181+0000","last_became_active":"2024-09-11T05:24:45.366531+0000","last_became_peered":"2024-09-11T05:24:45.366531+0000","last_unstale":"2024-09-11T06:04:34.567181+0000","last_undegraded":"2024-09-11T06:04:34.567181+0000","last_fullsized":"2024-09-11T06:04:34.567181+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-09-11T06:04:34.566892+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-09-11T06:04:34.566892+0000","last_clean_scrub_stamp":"2024-09-11T06:04:34.566892+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:05:05.482888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00102746,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"465'99","reported_seq":1075,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-09-11T06:04:33.610131+0000","last_change":"2024-09-11T06:04:33.610131+0000","last_active":"2024-09-11T06:04:33.610131+0000","last_peered":"2024-09-11T06:04:33.610131+0000","last_clean":"2024-09-11T06:04:33.610131+0000","last_became_active":"2024-09-11T05:24:40.651747+0000","last_became_peered":"2024-09-11T05:24:40.651747+0000","last_unstale":"2024-09-11T06:04:33.610131+0000","last_undegraded":"2024-09-11T06:04:33.610131+0000","last_fullsized":"2024-09-11T06:04:33.610131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'99","last_scrub_stamp":"2024-09-11T06:04:33.610024+0000","last_deep_scrub":"465'99","last_deep_scrub_stamp":"2024-09-11T06:04:33.610024+0000","last_clean_scrub_stamp":"2024-09-11T06:04:33.610024+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:49:08.040498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12546,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706153,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125096,"kb_used_data":3736,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246744,"statfs":{"total":96636764160,"available":96508665856,"internally_reserved":0,"allocated":3825664,"data_stored":2043425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706150,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147428,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94224412,"statfs":{"total":96636764160,"available":96485797888,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165476,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":94206364,"statfs":{"total":96636764160,"available":96467316736,"internally_reserved":0,"allocated":3887104,"data_stored":2633812,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6273,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:04:54.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-11T06:04:54.964 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T06:04:54.964 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T06:04:54.964 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-11T06:04:54.964 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:04:58.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-09-11T06:05:01.066 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-11T06:05:01.067 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-11T06:05:01.067 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:05:04.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-09-11T06:05:04.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-09-11T06:05:07.168 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-11T06:05:07.169 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-11T06:05:07.169 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:05:07.252 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-11T06:05:07.252 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T06:05:07.252 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-11T06:05:07.252 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:05:07.282 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-11T06:05:07.283 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T06:05:07.283 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-11T06:05:07.283 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:05:07.345 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-11T06:05:07.346 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T06:05:07.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:05:07.402 INFO:teuthology.orchestra.run.smithi067.stdout:2024-09-11T05:27:35.354983+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T06:05:07.402 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-11T06:05:07.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:05:07.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:05:07.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[WRN\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:05:07.567 INFO:teuthology.orchestra.run.smithi067.stdout:2024-09-11T05:27:35.354983+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-11T06:05:07.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T06:05:07.568 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-11T06:05:08.009 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T06:05:08.009 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-11T06:05:08.190 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T06:05:08.191 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-11T06:05:08.322 INFO:tasks.ceph:Archiving mon data... 2024-09-11T06:05:08.322 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/data/cluster1.mon.a.tgz 2024-09-11T06:05:08.323 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-11T06:05:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.9jfWW7LjFt 2024-09-11T06:05:08.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.9jfWW7LjFt 2024-09-11T06:05:08.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.9jfWW7LjFt 2024-09-11T06:05:08.549 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.9jfWW7LjFt is 411KB 2024-09-11T06:05:08.611 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.9jfWW7LjFt 2024-09-11T06:05:08.616 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T06:05:08.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:05:08.662 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:05:08.721 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T06:05:08.737 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T06:05:08.737 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T06:05:08.739 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi067/crash 2024-09-11T06:05:08.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T06:05:08.778 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi164/crash 2024-09-11T06:05:08.778 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T06:05:08.795 INFO:tasks.ceph:Compressing logs... 2024-09-11T06:05:08.796 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T06:05:08.822 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T06:05:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29877.log 2024-09-11T06:05:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28277.log 2024-09-11T06:05:08.838 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28439.log 2024-09-11T06:05:08.838 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29877.log.gz 2024-09-11T06:05:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28277.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28277.log.gz 2024-09-11T06:05:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-11T06:05:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-11T06:05:08.840 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28439.log: /var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28207.log 2024-09-11T06:05:08.840 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28439.log.gz 2024-09-11T06:05:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28589.log 2024-09-11T06:05:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28440.log 2024-09-11T06:05:08.841 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28207.log.gz 2024-09-11T06:05:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28362.log 2024-09-11T06:05:08.841 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28589.log.gz 2024-09-11T06:05:08.845 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29669.log 2024-09-11T06:05:08.845 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28440.log.gz 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25354.log 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28362.log.gz 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29669.log.gz 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29851.log 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.25354.log: /var/log/ceph/cluster1-client.admin.29825.log 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25354.log.gz 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29617.log 2024-09-11T06:05:08.846 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29851.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29721.log 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29825.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28519.log 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29617.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29721.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28232.log 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28519.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28232.log.gz 2024-09-11T06:05:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28303.log 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29901.log 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28518.log 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28303.log.gz 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29901.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29901.log.gz 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.28358.log 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28257.log 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28518.log.gz 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28358.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28358.log.gz 2024-09-11T06:05:08.851 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29773.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.28257.log.gz 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29773.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29773.log.gz 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29643.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27971.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29643.log.gz 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28329.log 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.27971.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27971.log.gz 2024-09-11T06:05:08.852 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28511.log 2024-09-11T06:05:08.855 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25295.log 2024-09-11T06:05:08.855 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28329.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25271.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28511.log: /var/log/ceph/cluster1-client.admin.25295.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25422.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25295.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.28511.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.25271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28363.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25271.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.25422.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29747.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.25422.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28363.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29591.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.28363.log.gz 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28615.log 2024-09-11T06:05:08.856 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29747.log.gz/var/log/ceph/cluster1-client.admin.29591.log: 2024-09-11T06:05:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25177.log 2024-09-11T06:05:08.857 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29591.log.gz 2024-09-11T06:05:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28615.log.gz 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28641.log 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.25177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25319.log 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25177.log.gz 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.28641.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28441.log 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.28641.log.gz 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.25319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29799.log 2024-09-11T06:05:08.859 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25319.log.gz/var/log/ceph/cluster1-client.admin.28441.log: 2024-09-11T06:05:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-11T06:05:08.860 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28441.log.gz 2024-09-11T06:05:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29799.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29799.log.gz 2024-09-11T06:05:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29695.log 2024-09-11T06:05:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-11T06:05:08.861 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-client.admin.29695.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21565.log 2024-09-11T06:05:08.861 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29695.log.gz 2024-09-11T06:05:08.861 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/ceph.tmp-client.admin.21565.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21565.log.gz 2024-09-11T06:05:08.864 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-11T06:05:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40253.log 2024-09-11T06:05:08.867 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37827.log 2024-09-11T06:05:08.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62211.log 2024-09-11T06:05:08.868 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40253.log.gz 2024-09-11T06:05:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38959.log 2024-09-11T06:05:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38799.log 2024-09-11T06:05:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38959.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62211.log.gz 2024-09-11T06:05:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54577.log 2024-09-11T06:05:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58838.log 2024-09-11T06:05:08.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42129.log 2024-09-11T06:05:08.872 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44871.log 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58838.log.gz 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/cluster1-client.mirror.42129.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54868.log 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44871.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42129.log.gz 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56375.log 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44871.log.gz 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54868.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/cluster2-client.mirror.54868.log.gz /var/log/ceph/cluster1-client.mirror.48591.log 2024-09-11T06:05:08.873 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37227.log 2024-09-11T06:05:08.875 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54577.log.gz 2024-09-11T06:05:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34460.log 2024-09-11T06:05:08.875 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48591.log.gz 2024-09-11T06:05:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63087.log 2024-09-11T06:05:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28789.log 2024-09-11T06:05:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63087.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34460.log.gz 2024-09-11T06:05:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63419.log 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56996.log 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28789.log: /var/log/ceph/cluster2-client.mirror.63419.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56375.log.gz 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28789.log.gz 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59963.log 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63419.log.gz 2024-09-11T06:05:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64623.log 2024-09-11T06:05:08.878 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48716.log 2024-09-11T06:05:08.878 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59963.log.gz 2024-09-11T06:05:08.878 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64623.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64623.log.gz 2024-09-11T06:05:08.879 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28739.log 2024-09-11T06:05:08.879 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56293.log 2024-09-11T06:05:08.880 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4%/var/log/ceph/cluster1-client.admin.28739.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56996.log.gz 2024-09-11T06:05:08.880 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28739.log.gz 2024-09-11T06:05:08.880 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48716.log.gz 2024-09-11T06:05:08.880 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32576.log 2024-09-11T06:05:08.881 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58793.log 2024-09-11T06:05:08.881 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56293.log: /var/log/ceph/cluster2-client.mirror.32576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53698.log 2024-09-11T06:05:08.882 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56293.log.gz 2024-09-11T06:05:08.882 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58793.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58793.log.gz 2024-09-11T06:05:08.883 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.42155.log 2024-09-11T06:05:08.883 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52758.log 2024-09-11T06:05:08.883 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53698.log.gz 2024-09-11T06:05:08.885 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.42155.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.42155.log.gz 2024-09-11T06:05:08.886 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32576.log.gz 2024-09-11T06:05:08.886 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61962.log 2024-09-11T06:05:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.63186.log 2024-09-11T06:05:08.887 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.61962.log: -- replaced with /var/log/ceph/cluster1-client.mirror.52758.log.gz 2024-09-11T06:05:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56062.log 2024-09-11T06:05:08.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51793.log 2024-09-11T06:05:08.888 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.63186.log: /var/log/ceph/cluster2-client.mirror.56062.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61962.log.gz 2024-09-11T06:05:08.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64098.log 2024-09-11T06:05:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51793.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51793.log.gz 2024-09-11T06:05:08.890 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56062.log.gz 2024-09-11T06:05:08.890 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.63186.log.gz 2024-09-11T06:05:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57389.log 2024-09-11T06:05:08.891 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62987.log 2024-09-11T06:05:08.891 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64098.log.gz 2024-09-11T06:05:08.891 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51841.log 2024-09-11T06:05:08.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55012.log 2024-09-11T06:05:08.892 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57389.log.gz/var/log/ceph/cluster2-client.mirror.62987.log: 2024-09-11T06:05:08.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51841.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62987.log.gz 87.7% 2024-09-11T06:05:08.893 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51841.log.gz 2024-09-11T06:05:08.893 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45974.log 2024-09-11T06:05:08.894 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50829.log 2024-09-11T06:05:08.894 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38821.log 2024-09-11T06:05:08.894 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.50829.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45974.log.gz 2024-09-11T06:05:08.894 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50829.log.gz 2024-09-11T06:05:08.895 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37985.log 2024-09-11T06:05:08.896 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38821.log: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44630.log 2024-09-11T06:05:08.896 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55012.log.gz 2024-09-11T06:05:08.897 INFO:teuthology.orchestra.run.smithi164.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38821.log.gz 2024-09-11T06:05:08.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36419.log 2024-09-11T06:05:08.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29286.log 2024-09-11T06:05:08.898 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36419.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44630.log.gz 2024-09-11T06:05:08.898 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.29025.log 2024-09-11T06:05:08.898 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.29286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45159.log 2024-09-11T06:05:08.899 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.29025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35966.log 2024-09-11T06:05:08.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45159.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45159.log.gz 2024-09-11T06:05:08.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47821.log 2024-09-11T06:05:08.903 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35966.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36419.log.gz 2024-09-11T06:05:08.904 INFO:teuthology.orchestra.run.smithi164.stderr: 66.9% -- replaced with /var/log/ceph/cluster1-client.admin.29286.log.gz 2024-09-11T06:05:08.907 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35966.log.gz 2024-09-11T06:05:08.908 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31396.log 2024-09-11T06:05:08.908 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48105.log 2024-09-11T06:05:08.909 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31396.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31396.log.gz 2024-09-11T06:05:08.909 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34412.log 2024-09-11T06:05:08.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48105.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47821.log.gz 2024-09-11T06:05:08.910 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52386.log 2024-09-11T06:05:08.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34412.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48105.log.gz 2024-09-11T06:05:08.912 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34412.log.gz 2024-09-11T06:05:08.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29508.log 2024-09-11T06:05:08.912 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56083.log 2024-09-11T06:05:08.913 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.29508.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52386.log.gz 2024-09-11T06:05:08.913 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57658.log 2024-09-11T06:05:08.913 INFO:teuthology.orchestra.run.smithi164.stderr: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.29508.log.gz 2024-09-11T06:05:08.914 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35092.log 2024-09-11T06:05:08.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56083.log: /var/log/ceph/cluster2-client.mirror.57658.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57658.log.gz 2024-09-11T06:05:08.915 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28279.log 2024-09-11T06:05:08.917 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35092.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56083.log.gz 2024-09-11T06:05:08.917 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57472.log 2024-09-11T06:05:08.918 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28279.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28279.log.gz 2024-09-11T06:05:08.918 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42861.log 2024-09-11T06:05:08.919 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57472.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57472.log.gz 2024-09-11T06:05:08.920 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34662.log 2024-09-11T06:05:08.921 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42861.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42861.log.gz 2024-09-11T06:05:08.922 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56000.log 2024-09-11T06:05:08.923 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.34662.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34662.log.gz 2024-09-11T06:05:08.923 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56603.log 2024-09-11T06:05:08.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56000.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.29025.log.gz 2024-09-11T06:05:08.930 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56000.log.gz 2024-09-11T06:05:08.936 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41035.log 2024-09-11T06:05:08.938 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56603.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56603.log.gz 2024-09-11T06:05:08.938 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38167.log 2024-09-11T06:05:08.939 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41035.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41035.log.gz 2024-09-11T06:05:08.939 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48017.log 2024-09-11T06:05:08.940 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38167.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38167.log.gz 2024-09-11T06:05:08.940 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58523.log 2024-09-11T06:05:08.941 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48017.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48017.log.gz 2024-09-11T06:05:08.942 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62922.log 2024-09-11T06:05:08.942 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58523.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58523.log.gz 2024-09-11T06:05:08.943 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59918.log 2024-09-11T06:05:08.944 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62922.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62922.log.gz 2024-09-11T06:05:08.944 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50586.log 2024-09-11T06:05:08.945 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59918.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59918.log.gz 2024-09-11T06:05:08.945 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45802.log 2024-09-11T06:05:08.948 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50586.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50586.log.gz 2024-09-11T06:05:08.948 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33010.log 2024-09-11T06:05:08.950 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45802.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45802.log.gz 2024-09-11T06:05:08.951 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44726.log 2024-09-11T06:05:08.957 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33010.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33010.log.gz 2024-09-11T06:05:08.957 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28864.log 2024-09-11T06:05:08.958 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44726.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44726.log.gz 2024-09-11T06:05:08.959 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46364.log 2024-09-11T06:05:08.959 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28864.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28864.log.gz 2024-09-11T06:05:08.960 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58568.log 2024-09-11T06:05:08.961 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46364.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46364.log.gz 2024-09-11T06:05:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33030.log 2024-09-11T06:05:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58568.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58568.log.gz 2024-09-11T06:05:08.963 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57203.log 2024-09-11T06:05:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33030.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33030.log.gz 2024-09-11T06:05:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34157.log 2024-09-11T06:05:08.971 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57203.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57203.log.gz 2024-09-11T06:05:08.971 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31444.log 2024-09-11T06:05:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.34157.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34157.log.gz 2024-09-11T06:05:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37544.log 2024-09-11T06:05:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31444.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31444.log.gz 2024-09-11T06:05:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63159.log 2024-09-11T06:05:08.975 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37544.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37544.log.gz 2024-09-11T06:05:08.975 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53746.log 2024-09-11T06:05:08.978 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63159.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63159.log.gz 2024-09-11T06:05:08.978 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51721.log 2024-09-11T06:05:08.979 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53746.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53746.log.gz 2024-09-11T06:05:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57575.log 2024-09-11T06:05:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51721.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51721.log.gz 2024-09-11T06:05:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56768.log 2024-09-11T06:05:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57575.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57575.log.gz 2024-09-11T06:05:08.983 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28985.log 2024-09-11T06:05:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56768.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56768.log.gz 2024-09-11T06:05:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50015.log 2024-09-11T06:05:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.28985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51380.log 2024-09-11T06:05:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50015.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50015.log.gz 2024-09-11T06:05:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29542.log 2024-09-11T06:05:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51380.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51380.log.gz 2024-09-11T06:05:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31237.log 2024-09-11T06:05:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29542.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29542.log.gz 2024-09-11T06:05:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59153.log 2024-09-11T06:05:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31237.log: 74.5% -- replaced with /var/log/ceph/cluster1-client.mirror.31237.log.gz 2024-09-11T06:05:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49885.log 2024-09-11T06:05:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59153.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59153.log.gz 2024-09-11T06:05:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42268.log 2024-09-11T06:05:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49885.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49885.log.gz 2024-09-11T06:05:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47660.log 2024-09-11T06:05:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42268.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42268.log.gz 2024-09-11T06:05:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52469.log 2024-09-11T06:05:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47660.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47660.log.gz 2024-09-11T06:05:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59805.log 2024-09-11T06:05:08.999 INFO:teuthology.orchestra.run.smithi164.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28985.log.gz 2024-09-11T06:05:08.999 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52469.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52469.log.gz 2024-09-11T06:05:09.000 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61683.log 2024-09-11T06:05:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59805.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59805.log.gz 2024-09-11T06:05:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36977.log 2024-09-11T06:05:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61683.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61683.log.gz 2024-09-11T06:05:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28360.log 2024-09-11T06:05:09.004 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36977.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36977.log.gz 2024-09-11T06:05:09.004 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54909.log 2024-09-11T06:05:09.005 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28360.log.gz 2024-09-11T06:05:09.005 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34184.log 2024-09-11T06:05:09.006 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54909.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54909.log.gz 2024-09-11T06:05:09.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56955.log 2024-09-11T06:05:09.008 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.34184.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34184.log.gz 2024-09-11T06:05:09.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59468.log 2024-09-11T06:05:09.009 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56955.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56955.log.gz 2024-09-11T06:05:09.010 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53148.log 2024-09-11T06:05:09.011 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59468.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59468.log.gz 2024-09-11T06:05:09.011 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54971.log 2024-09-11T06:05:09.013 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53148.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53148.log.gz 2024-09-11T06:05:09.013 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50184.log 2024-09-11T06:05:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54971.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54971.log.gz 2024-09-11T06:05:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28049.log 2024-09-11T06:05:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50184.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50184.log.gz 2024-09-11T06:05:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63950.log 2024-09-11T06:05:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28049.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28049.log.gz 2024-09-11T06:05:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.28981.log 2024-09-11T06:05:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63950.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63950.log.gz 2024-09-11T06:05:09.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-09-11T06:05:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49579.log 2024-09-11T06:05:09.019 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.28981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55455.log 2024-09-11T06:05:09.020 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49579.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49579.log.gz 2024-09-11T06:05:09.020 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37705.log 2024-09-11T06:05:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55455.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55455.log.gz 2024-09-11T06:05:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63186.log 2024-09-11T06:05:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37705.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.28981.log.gz 2024-09-11T06:05:09.025 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37705.log.gz 2024-09-11T06:05:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46480.log 2024-09-11T06:05:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63186.log: 78.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63186.log.gz 2024-09-11T06:05:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45468.log 2024-09-11T06:05:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46480.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46480.log.gz 2024-09-11T06:05:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38571.log 2024-09-11T06:05:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45468.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45468.log.gz 2024-09-11T06:05:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35141.log 2024-09-11T06:05:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.38571.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38571.log.gz 2024-09-11T06:05:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36857.log 2024-09-11T06:05:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.35141.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.35141.log.gz 2024-09-11T06:05:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53722.log 2024-09-11T06:05:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36857.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36857.log.gz 2024-09-11T06:05:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33052.log 2024-09-11T06:05:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53722.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53722.log.gz 2024-09-11T06:05:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52493.log 2024-09-11T06:05:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33052.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33052.log.gz 2024-09-11T06:05:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28509.log 2024-09-11T06:05:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52493.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52493.log.gz 2024-09-11T06:05:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61219.log 2024-09-11T06:05:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28509.log.gz 2024-09-11T06:05:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62789.log 2024-09-11T06:05:09.039 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61219.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61219.log.gz 2024-09-11T06:05:09.039 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33890.log 2024-09-11T06:05:09.040 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62789.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62789.log.gz 2024-09-11T06:05:09.041 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48760.log 2024-09-11T06:05:09.042 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33890.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33890.log.gz 2024-09-11T06:05:09.042 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.28921.log 2024-09-11T06:05:09.043 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48760.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48760.log.gz 2024-09-11T06:05:09.043 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30380.log 2024-09-11T06:05:09.044 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.28921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40462.log 2024-09-11T06:05:09.045 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.30380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54397.log 2024-09-11T06:05:09.046 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40462.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40462.log.gz 2024-09-11T06:05:09.048 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28517.log 2024-09-11T06:05:09.049 INFO:teuthology.orchestra.run.smithi164.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.28921.log.gz 2024-09-11T06:05:09.050 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54397.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54397.log.gz 2024-09-11T06:05:09.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38740.log 2024-09-11T06:05:09.051 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28517.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28517.log.gz 2024-09-11T06:05:09.051 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49073.log 2024-09-11T06:05:09.052 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38740.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38740.log.gz 2024-09-11T06:05:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52323.log 2024-09-11T06:05:09.054 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49073.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49073.log.gz 2024-09-11T06:05:09.054 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61727.log 2024-09-11T06:05:09.055 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52323.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52323.log.gz 2024-09-11T06:05:09.055 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37185.log 2024-09-11T06:05:09.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61727.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61727.log.gz 2024-09-11T06:05:09.087 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-11T06:05:09.088 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47863.log 2024-09-11T06:05:09.090 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37185.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37185.log.gz 2024-09-11T06:05:09.096 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41405.log 2024-09-11T06:05:09.097 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47863.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47863.log.gz 2024-09-11T06:05:09.104 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51745.log 2024-09-11T06:05:09.105 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41405.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41405.log.gz 2024-09-11T06:05:09.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39628.log 2024-09-11T06:05:09.125 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51745.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51745.log.gz 2024-09-11T06:05:09.132 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51046.log 2024-09-11T06:05:09.140 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40971.log 2024-09-11T06:05:09.141 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51046.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51046.log.gz 2024-09-11T06:05:09.142 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54249.log 2024-09-11T06:05:09.143 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40971.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40971.log.gz 2024-09-11T06:05:09.148 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57306.log 2024-09-11T06:05:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39274.log 2024-09-11T06:05:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57306.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57306.log.gz 2024-09-11T06:05:09.166 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54249.log.gz 2024-09-11T06:05:09.169 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31879.log 2024-09-11T06:05:09.169 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39274.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39274.log.gz 2024-09-11T06:05:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54620.log 2024-09-11T06:05:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31879.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31879.log.gz 2024-09-11T06:05:09.195 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-11T06:05:09.201 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51219.log 2024-09-11T06:05:09.201 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54620.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54620.log.gz 2024-09-11T06:05:09.221 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61541.log 2024-09-11T06:05:09.222 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51219.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51219.log.gz 2024-09-11T06:05:09.240 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58928.log 2024-09-11T06:05:09.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61541.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61541.log.gz 2024-09-11T06:05:09.245 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63087.log.gz 2024-09-11T06:05:09.252 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46808.log 2024-09-11T06:05:09.253 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58928.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58928.log.gz 2024-09-11T06:05:09.257 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38799.log.gz 2024-09-11T06:05:09.257 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53481.log 2024-09-11T06:05:09.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46808.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46808.log.gz 2024-09-11T06:05:09.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57782.log 2024-09-11T06:05:09.259 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53481.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53481.log.gz 2024-09-11T06:05:09.260 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56810.log 2024-09-11T06:05:09.260 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57782.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57782.log.gz 2024-09-11T06:05:09.261 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.37373.log 2024-09-11T06:05:09.264 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56810.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56810.log.gz 2024-09-11T06:05:09.264 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43666.log 2024-09-11T06:05:09.267 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.37373.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37227.log.gz 2024-09-11T06:05:09.276 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41585.log 2024-09-11T06:05:09.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43666.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43666.log.gz 2024-09-11T06:05:09.278 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64415.log 2024-09-11T06:05:09.295 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41585.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35092.log.gz 2024-09-11T06:05:09.296 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51621.log 2024-09-11T06:05:09.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64415.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64415.log.gz 2024-09-11T06:05:09.297 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38917.log 2024-09-11T06:05:09.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51621.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.51621.log.gz 2024-09-11T06:05:09.300 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47077.log 2024-09-11T06:05:09.301 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38917.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38917.log.gz 2024-09-11T06:05:09.302 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55248.log 2024-09-11T06:05:09.303 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47077.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47077.log.gz 2024-09-11T06:05:09.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41195.log 2024-09-11T06:05:09.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55248.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55248.log.gz 2024-09-11T06:05:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35903.log 2024-09-11T06:05:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41195.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41195.log.gz 2024-09-11T06:05:09.306 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43232.log 2024-09-11T06:05:09.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35903.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35903.log.gz 2024-09-11T06:05:09.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40743.log 2024-09-11T06:05:09.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43232.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43232.log.gz 2024-09-11T06:05:09.314 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40660.log 2024-09-11T06:05:09.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.40743.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40743.log.gz 2024-09-11T06:05:09.317 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53939.log 2024-09-11T06:05:09.319 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40660.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40660.log.gz 2024-09-11T06:05:09.319 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29627.log 2024-09-11T06:05:09.320 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53939.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53939.log.gz 2024-09-11T06:05:09.320 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31162.log 2024-09-11T06:05:09.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49531.log 2024-09-11T06:05:09.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.31162.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29627.log.gz 2024-09-11T06:05:09.323 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.31162.log.gz 2024-09-11T06:05:09.324 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.31237.log 2024-09-11T06:05:09.324 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49531.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49531.log.gz 2024-09-11T06:05:09.324 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43015.log 2024-09-11T06:05:09.326 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.31237.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.31237.log.gz 2024-09-11T06:05:09.327 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63361.log 2024-09-11T06:05:09.328 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43015.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43015.log.gz 2024-09-11T06:05:09.328 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52123.log 2024-09-11T06:05:09.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63361.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63361.log.gz 2024-09-11T06:05:09.331 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34635.log 2024-09-11T06:05:09.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52123.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52123.log.gz 2024-09-11T06:05:09.333 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45203.log 2024-09-11T06:05:09.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.34635.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.34635.log.gz 2024-09-11T06:05:09.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35726.log 2024-09-11T06:05:09.335 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45203.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45203.log.gz 2024-09-11T06:05:09.335 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43060.log 2024-09-11T06:05:09.341 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35726.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35726.log.gz 2024-09-11T06:05:09.342 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37098.log 2024-09-11T06:05:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43060.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43060.log.gz 2024-09-11T06:05:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36525.log 2024-09-11T06:05:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37098.log: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37098.log.gz 2024-09-11T06:05:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33471.log 2024-09-11T06:05:09.354 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36525.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36525.log.gz 2024-09-11T06:05:09.354 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36204.log 2024-09-11T06:05:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33471.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33471.log.gz 2024-09-11T06:05:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64169.log 2024-09-11T06:05:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36204.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36204.log.gz 2024-09-11T06:05:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32134.log 2024-09-11T06:05:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.64169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64169.log.gz 2024-09-11T06:05:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34892.log 2024-09-11T06:05:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.32134.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.32134.log.gz 2024-09-11T06:05:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63777.log 2024-09-11T06:05:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34892.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34892.log.gz 2024-09-11T06:05:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54599.log 2024-09-11T06:05:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63777.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63777.log.gz 2024-09-11T06:05:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35588.log 2024-09-11T06:05:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54599.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54599.log.gz 2024-09-11T06:05:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35608.log 2024-09-11T06:05:09.374 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35588.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35588.log.gz 2024-09-11T06:05:09.375 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36654.log 2024-09-11T06:05:09.375 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35608.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35608.log.gz 2024-09-11T06:05:09.376 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32639.log 2024-09-11T06:05:09.382 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36654.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36654.log.gz 2024-09-11T06:05:09.382 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56706.log 2024-09-11T06:05:09.388 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32639.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.32639.log.gz 2024-09-11T06:05:09.388 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55558.log 2024-09-11T06:05:09.390 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56706.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56706.log.gz 2024-09-11T06:05:09.390 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44767.log 2024-09-11T06:05:09.391 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55558.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55558.log.gz 2024-09-11T06:05:09.391 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48567.log 2024-09-11T06:05:09.393 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44767.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44767.log.gz 2024-09-11T06:05:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38446.log 2024-09-11T06:05:09.395 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48567.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48567.log.gz 2024-09-11T06:05:09.395 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50918.log 2024-09-11T06:05:09.396 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38446.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38446.log.gz 2024-09-11T06:05:09.396 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33866.log 2024-09-11T06:05:09.398 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50918.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50918.log.gz 2024-09-11T06:05:09.399 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48495.log 2024-09-11T06:05:09.400 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33866.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33866.log.gz 2024-09-11T06:05:09.400 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39486.log 2024-09-11T06:05:09.401 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48495.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48495.log.gz 2024-09-11T06:05:09.401 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46085.log 2024-09-11T06:05:09.402 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39486.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39486.log.gz 2024-09-11T06:05:09.403 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58748.log 2024-09-11T06:05:09.404 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46085.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46085.log.gz 2024-09-11T06:05:09.404 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28919.log 2024-09-11T06:05:09.405 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58748.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58748.log.gz 2024-09-11T06:05:09.411 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31352.log 2024-09-11T06:05:09.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.28919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48832.log 2024-09-11T06:05:09.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31352.log: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.28919.log.gz 2024-09-11T06:05:09.411 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31352.log.gz 2024-09-11T06:05:09.412 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.32300.log 2024-09-11T06:05:09.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48832.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48832.log.gz 2024-09-11T06:05:09.412 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51263.log 2024-09-11T06:05:09.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.32300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.33655.log 2024-09-11T06:05:09.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51263.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51263.log.gz 2024-09-11T06:05:09.436 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42620.log 2024-09-11T06:05:09.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.33655.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.33655.log.gz 2024-09-11T06:05:09.444 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62663.log 2024-09-11T06:05:09.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42620.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42620.log.gz 2024-09-11T06:05:09.452 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39765.log 2024-09-11T06:05:09.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62663.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62663.log.gz 2024-09-11T06:05:09.460 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55331.log 2024-09-11T06:05:09.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39765.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39765.log.gz 2024-09-11T06:05:09.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59558.log 2024-09-11T06:05:09.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55331.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55331.log.gz 2024-09-11T06:05:09.471 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37827.log.gz 2024-09-11T06:05:09.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55765.log 2024-09-11T06:05:09.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59558.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59558.log.gz 2024-09-11T06:05:09.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38506.log 2024-09-11T06:05:09.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55765.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55765.log.gz 2024-09-11T06:05:09.474 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63065.log 2024-09-11T06:05:09.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38506.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38506.log.gz 2024-09-11T06:05:09.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42598.log 2024-09-11T06:05:09.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63065.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63065.log.gz 2024-09-11T06:05:09.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31996.log 2024-09-11T06:05:09.478 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42598.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42598.log.gz 2024-09-11T06:05:09.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44437.log 2024-09-11T06:05:09.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31996.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31996.log.gz 2024-09-11T06:05:09.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37624.log 2024-09-11T06:05:09.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44437.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44437.log.gz 2024-09-11T06:05:09.486 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33310.log 2024-09-11T06:05:09.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37624.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37624.log.gz 2024-09-11T06:05:09.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45104.log 2024-09-11T06:05:09.488 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33310.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33310.log.gz 2024-09-11T06:05:09.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34364.log 2024-09-11T06:05:09.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45104.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45104.log.gz 2024-09-11T06:05:09.490 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36333.log 2024-09-11T06:05:09.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34364.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34364.log.gz 2024-09-11T06:05:09.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33494.log 2024-09-11T06:05:09.497 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36333.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36333.log.gz 2024-09-11T06:05:09.497 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62317.log 2024-09-11T06:05:09.502 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33494.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41585.log.gz 2024-09-11T06:05:09.503 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33494.log.gz 2024-09-11T06:05:09.504 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.31555.log 2024-09-11T06:05:09.504 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44265.log 2024-09-11T06:05:09.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62317.log: /var/log/ceph/cluster1-client.mirror.0.31555.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62317.log.gz 2024-09-11T06:05:09.505 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42533.log 2024-09-11T06:05:09.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44265.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44265.log.gz 2024-09-11T06:05:09.507 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49359.log 2024-09-11T06:05:09.508 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42533.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42533.log.gz 2024-09-11T06:05:09.508 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47270.log 2024-09-11T06:05:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49359.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49359.log.gz 2024-09-11T06:05:09.512 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.31555.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52541.log 2024-09-11T06:05:09.512 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47270.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47270.log.gz 2024-09-11T06:05:09.512 INFO:teuthology.orchestra.run.smithi164.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.37373.log.gz 2024-09-11T06:05:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.39068.log 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64493.log 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52541.log: /var/log/ceph/cluster1-client.mirror.1.39068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56396.log 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64493.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52541.log.gz 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64493.log.gz 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36482.log 2024-09-11T06:05:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32209.log 2024-09-11T06:05:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36482.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56396.log.gz 2024-09-11T06:05:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.28985.log 2024-09-11T06:05:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32209.log: 74.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32209.log.gz 2024-09-11T06:05:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47776.log 2024-09-11T06:05:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.28985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47222.log 2024-09-11T06:05:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47776.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47776.log.gz 2024-09-11T06:05:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45227.log 2024-09-11T06:05:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47222.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47222.log.gz 2024-09-11T06:05:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58005.log 2024-09-11T06:05:09.520 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36482.log.gz 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57368.log 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.58005.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45227.log.gz 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58005.log.gz 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54785.log 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.55869.log 2024-09-11T06:05:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57368.log: /var/log/ceph/cluster2-client.mirror.54785.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54785.log.gz 2024-09-11T06:05:09.523 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57368.log.gz 2024-09-11T06:05:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31975.log 2024-09-11T06:05:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63970.log 2024-09-11T06:05:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.55869.log: /var/log/ceph/cluster1-client.mirror.31975.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.55869.log.gz 2024-09-11T06:05:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41643.log 2024-09-11T06:05:09.527 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.39068.log.gz 2024-09-11T06:05:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63970.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38959.log.gz 2024-09-11T06:05:09.529 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63970.log.gz 2024-09-11T06:05:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59670.log 2024-09-11T06:05:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60510.log 2024-09-11T06:05:09.537 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59670.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41643.log.gz 2024-09-11T06:05:09.537 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28283.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59670.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57616.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60510.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31975.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28283.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28283.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50425.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.32300.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41912.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57616.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60510.log.gz/var/log/ceph/cluster2-client.mirror.50425.log: 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52999.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41912.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50425.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58770.log 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41912.log.gz 2024-09-11T06:05:09.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51094.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44044.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52999.log.gz 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37847.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58770.log.gz 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51094.log.gz 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31647.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62832.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37847.log: /var/log/ceph/cluster2-client.mirror.31647.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37847.log.gz 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57616.log.gz 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31034.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58635.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62832.log: /var/log/ceph/cluster1-client.mirror.31034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62296.log 2024-09-11T06:05:09.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58635.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62832.log.gz 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44044.log.gz 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56727.log 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58635.log.gz 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44548.log 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34020.log 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41891.log 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.44548.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56727.log.gz 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34020.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44548.log.gz 2024-09-11T06:05:09.540 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34020.log.gz 2024-09-11T06:05:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.60623.log 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29491.log 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31034.log.gz 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62296.log.gz 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41891.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41891.log.gz 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.60623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61749.log 2024-09-11T06:05:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.29491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41173.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38757.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr: 83.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61749.log.gz 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41173.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31647.log.gz 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52275.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32507.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52275.log: 87.7% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38757.log.gz 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52275.log.gz 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44654.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25333.log 2024-09-11T06:05:09.547 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4%/var/log/ceph/cluster1-client.mirror.44654.log: -- replaced with /var/log/ceph/cluster1-client.mirror.32507.log.gz 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51552.log 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44654.log.gz 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58680.log 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25333.log: /var/log/ceph/cluster1-client.mirror.51552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25333.log.gz 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51552.log.gz 2024-09-11T06:05:09.548 INFO:teuthology.orchestra.run.smithi164.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.29491.log.gz 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61706.log 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51986.log 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41764.log 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58680.log.gz 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61706.log: /var/log/ceph/cluster1-client.mirror.51986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50164.log 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41764.log: 87.5% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51986.log.gz 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61706.log.gz 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41764.log.gz 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35923.log 2024-09-11T06:05:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-11T06:05:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54806.log 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.1.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50164.log.gz 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46504.log 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28431.log 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54806.log: /var/log/ceph/cluster1-client.mirror.46504.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54806.log.gz 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46504.log.gz 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48656.log 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57870.log 2024-09-11T06:05:09.554 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28431.log.gz 2024-09-11T06:05:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59738.log 2024-09-11T06:05:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48656.log: /var/log/ceph/cluster1-client.mirror.57870.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57870.log.gz 2024-09-11T06:05:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54469.log 2024-09-11T06:05:09.557 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59738.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59738.log.gz 2024-09-11T06:05:09.557 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56313.log 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48656.log.gz 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54469.log: 92.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42658.log 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.60623.log.gz 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54469.log.gz 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56313.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35923.log.gz 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58545.log 2024-09-11T06:05:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31492.log 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58545.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58545.log.gz 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42658.log.gz 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56976.log 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44506.log 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58118.log 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31492.log.gz 2024-09-11T06:05:09.563 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44506.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56313.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56976.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38571.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.28985.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62488.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58118.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.38571.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56582.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53550.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 85.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62488.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.56582.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44506.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46191.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54490.log 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56582.log.gz 2024-09-11T06:05:09.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53550.log: /var/log/ceph/cluster1-client.mirror.46191.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46191.log.gz 2024-09-11T06:05:09.567 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53550.log.gz 2024-09-11T06:05:09.567 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29269.log 2024-09-11T06:05:09.567 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58883.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.29269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46408.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38571.log.gz 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58883.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58883.log.gz 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59940.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38066.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59940.log: gzip 87.7% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.46408.log.gz 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.47559.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59940.log.gz/var/log/ceph/cluster2-client.mirror.38066.log: 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63740.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63476.log 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37985.log.gz 2024-09-11T06:05:09.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50449.log 2024-09-11T06:05:09.569 INFO:teuthology.orchestra.run.smithi164.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.29269.log.gz 2024-09-11T06:05:09.569 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47559.log.gz 2024-09-11T06:05:09.569 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63740.log.gz 2024-09-11T06:05:09.569 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63476.log: 82.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63476.log.gz 2024-09-11T06:05:09.581 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30938.log 2024-09-11T06:05:09.581 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63028.log 2024-09-11T06:05:09.581 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44967.log 2024-09-11T06:05:09.581 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.30938.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50449.log.gz 2024-09-11T06:05:09.581 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31689.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63028.log.gz 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61407.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57938.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44967.log.gz 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63045.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 85.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61407.log.gz 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53915.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.31689.log.gz 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57938.log.gz 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44847.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48856.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53915.log: /var/log/ceph/cluster2-client.mirror.44847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64571.log 2024-09-11T06:05:09.582 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53915.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63045.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.48856.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44847.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48856.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45540.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64571.log: gzip -5 --verbose -- 87.5% /var/log/ceph/cluster1-client.mirror.46239.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30938.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64571.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37334.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46304.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45540.log.gz 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46239.log: /var/log/ceph/cluster2-client.mirror.37334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47511.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37334.log.gz/var/log/ceph/cluster1-client.mirror.46304.log: 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50367.log 2024-09-11T06:05:09.583 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46239.log.gz 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60098.log 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38468.log 87.6% 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47511.log.gz 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43738.log 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47294.log 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60098.log.gz 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50367.log.gz 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38468.log: /var/log/ceph/cluster1-client.mirror.43738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56644.log 2024-09-11T06:05:09.584 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43738.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53433.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47600.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47294.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56644.log: /var/log/ceph/cluster1-client.mirror.53433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36697.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53433.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5%/var/log/ceph/cluster1-client.mirror.47600.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56644.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46304.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36697.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47600.log.gz 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28435.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38125.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28435.log: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31119.log 2024-09-11T06:05:09.585 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36697.log.gz 0.0% 2024-09-11T06:05:09.586 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.28435.log.gz 2024-09-11T06:05:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39569.log 2024-09-11T06:05:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28955.log 2024-09-11T06:05:09.586 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.31119.log: -- replaced with /var/log/ceph/cluster2-client.mirror.38125.log.gz 2024-09-11T06:05:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39569.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39569.log.gz 2024-09-11T06:05:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40521.log 2024-09-11T06:05:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.28955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40991.log 2024-09-11T06:05:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34612.log 2024-09-11T06:05:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43690.log 2024-09-11T06:05:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34612.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40991.log.gz 2024-09-11T06:05:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42702.log 2024-09-11T06:05:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43690.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43690.log.gz 2024-09-11T06:05:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55413.log 2024-09-11T06:05:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42702.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42702.log.gz 2024-09-11T06:05:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53110.log 2024-09-11T06:05:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55413.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55413.log.gz 2024-09-11T06:05:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56231.log 2024-09-11T06:05:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57893.log 2024-09-11T06:05:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53110.log: /var/log/ceph/cluster2-client.mirror.56231.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56231.log.gz 2024-09-11T06:05:09.593 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53110.log.gz 2024-09-11T06:05:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35504.log 2024-09-11T06:05:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46215.log 2024-09-11T06:05:09.595 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57893.log.gz 2024-09-11T06:05:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44413.log 2024-09-11T06:05:09.605 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46215.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46215.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40142.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35504.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63908.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44413.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40142.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54490.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46914.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63908.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63908.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40142.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40026.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40420.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46914.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49118.log 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40420.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40420.log.gz 2024-09-11T06:05:09.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37564.log 2024-09-11T06:05:09.607 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49118.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49118.log.gz 2024-09-11T06:05:09.607 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45844.log 2024-09-11T06:05:09.608 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37564.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37564.log.gz 2024-09-11T06:05:09.608 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36032.log 2024-09-11T06:05:09.610 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45844.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45844.log.gz 2024-09-11T06:05:09.610 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58095.log 2024-09-11T06:05:09.616 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36032.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36032.log.gz 2024-09-11T06:05:09.616 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45444.log 2024-09-11T06:05:09.617 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58095.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58095.log.gz 2024-09-11T06:05:09.617 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38548.log 2024-09-11T06:05:09.619 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45444.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45444.log.gz 2024-09-11T06:05:09.619 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38387.log 2024-09-11T06:05:09.628 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31903.log 2024-09-11T06:05:09.631 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38387.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39628.log.gz 2024-09-11T06:05:09.644 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54445.log 2024-09-11T06:05:09.645 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31903.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31903.log.gz 2024-09-11T06:05:09.652 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52344.log 2024-09-11T06:05:09.653 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54445.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54445.log.gz 2024-09-11T06:05:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34590.log 2024-09-11T06:05:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52344.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52344.log.gz 2024-09-11T06:05:09.685 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51769.log 2024-09-11T06:05:09.686 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34590.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34590.log.gz 2024-09-11T06:05:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42485.log 2024-09-11T06:05:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51769.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51769.log.gz 2024-09-11T06:05:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39375.log 2024-09-11T06:05:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42485.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42485.log.gz 2024-09-11T06:05:09.728 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43160.log 2024-09-11T06:05:09.744 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63998.log 2024-09-11T06:05:09.745 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43160.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43160.log.gz 2024-09-11T06:05:09.761 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63382.log 2024-09-11T06:05:09.761 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63998.log.gz 2024-09-11T06:05:09.769 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35300.log 2024-09-11T06:05:09.769 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63382.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63382.log.gz 2024-09-11T06:05:09.784 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37607.log 2024-09-11T06:05:09.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35300.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35300.log.gz 2024-09-11T06:05:09.801 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54847.log 2024-09-11T06:05:09.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37607.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37607.log.gz 2024-09-11T06:05:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41849.log 2024-09-11T06:05:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54847.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54847.log.gz 2024-09-11T06:05:09.832 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40874.log 2024-09-11T06:05:09.833 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41849.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41849.log.gz 2024-09-11T06:05:09.844 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31372.log 2024-09-11T06:05:09.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40874.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40874.log.gz 2024-09-11T06:05:09.860 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47993.log 2024-09-11T06:05:09.861 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31372.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31372.log.gz 2024-09-11T06:05:09.877 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29525.log 2024-09-11T06:05:09.877 INFO:teuthology.orchestra.run.smithi164.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.28955.log.gz 2024-09-11T06:05:09.877 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47993.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47993.log.gz 2024-09-11T06:05:09.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55054.log 2024-09-11T06:05:09.895 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29525.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29525.log.gz 2024-09-11T06:05:09.902 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38387.log.gz 2024-09-11T06:05:09.902 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54108.log 2024-09-11T06:05:09.903 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55054.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55054.log.gz 2024-09-11T06:05:09.903 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61364.log 2024-09-11T06:05:09.904 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54108.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54108.log.gz 2024-09-11T06:05:09.904 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45603.log 2024-09-11T06:05:09.905 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61364.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61364.log.gz 2024-09-11T06:05:09.905 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36837.log 2024-09-11T06:05:09.907 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45603.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45603.log.gz 2024-09-11T06:05:09.907 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47178.log 2024-09-11T06:05:09.908 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36837.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36837.log.gz 2024-09-11T06:05:09.909 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.35768.log 2024-09-11T06:05:09.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47178.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47178.log.gz 2024-09-11T06:05:09.910 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42967.log 2024-09-11T06:05:09.913 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.35768.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.3.35768.log.gz 2024-09-11T06:05:09.913 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50084.log 2024-09-11T06:05:09.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42967.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42967.log.gz 2024-09-11T06:05:09.914 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53811.log 2024-09-11T06:05:09.917 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50084.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50084.log.gz 2024-09-11T06:05:09.917 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57534.log 2024-09-11T06:05:09.919 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53811.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53811.log.gz 2024-09-11T06:05:09.919 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32435.log 2024-09-11T06:05:09.920 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57534.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57534.log.gz 2024-09-11T06:05:09.921 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63138.log 2024-09-11T06:05:09.922 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32435.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32435.log.gz 2024-09-11T06:05:09.922 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48957.log 2024-09-11T06:05:09.923 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63138.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63138.log.gz 2024-09-11T06:05:09.923 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40046.log 2024-09-11T06:05:09.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48957.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48957.log.gz 2024-09-11T06:05:09.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51882.log 2024-09-11T06:05:09.926 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40046.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40046.log.gz 2024-09-11T06:05:09.926 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39728.log 2024-09-11T06:05:09.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51882.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51882.log.gz 2024-09-11T06:05:09.929 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46432.log 2024-09-11T06:05:09.929 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39728.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39728.log.gz 2024-09-11T06:05:09.930 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34210.log 2024-09-11T06:05:09.930 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46432.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46432.log.gz 2024-09-11T06:05:09.931 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41279.log 2024-09-11T06:05:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34210.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.34210.log.gz 2024-09-11T06:05:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43039.log 2024-09-11T06:05:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41279.log: 84.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41279.log.gz 2024-09-11T06:05:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33334.log 2024-09-11T06:05:09.935 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43039.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43039.log.gz 2024-09-11T06:05:09.935 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50545.log 2024-09-11T06:05:09.936 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33334.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33334.log.gz 2024-09-11T06:05:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53987.log 2024-09-11T06:05:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50545.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50545.log.gz 2024-09-11T06:05:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61582.log 2024-09-11T06:05:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53987.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53987.log.gz 2024-09-11T06:05:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32760.log 2024-09-11T06:05:09.942 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61582.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61582.log.gz 2024-09-11T06:05:09.942 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44461.log 2024-09-11T06:05:09.944 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32760.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32760.log.gz 2024-09-11T06:05:09.945 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39708.log 2024-09-11T06:05:09.946 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44461.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44461.log.gz 2024-09-11T06:05:09.946 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34710.log 2024-09-11T06:05:09.951 INFO:teuthology.orchestra.run.smithi067.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-11T06:05:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39296.log 2024-09-11T06:05:09.965 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34710.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34710.log.gz 2024-09-11T06:05:09.968 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52082.log 2024-09-11T06:05:09.969 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39296.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40026.log.gz 2024-09-11T06:05:09.978 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41173.log.gz 2024-09-11T06:05:09.988 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37725.log 2024-09-11T06:05:09.989 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52082.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52082.log.gz 2024-09-11T06:05:09.996 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50898.log 2024-09-11T06:05:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37725.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37725.log.gz 2024-09-11T06:05:10.004 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36181.log 2024-09-11T06:05:10.006 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50898.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50898.log.gz 2024-09-11T06:05:10.017 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43080.log 2024-09-11T06:05:10.033 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36181.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41623.log 2024-09-11T06:05:10.034 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43080.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43080.log.gz 2024-09-11T06:05:10.036 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36181.log.gz 2024-09-11T06:05:10.048 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58163.log 2024-09-11T06:05:10.050 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41623.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41623.log.gz 2024-09-11T06:05:10.064 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55806.log 2024-09-11T06:05:10.065 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58163.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58163.log.gz 2024-09-11T06:05:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56541.log 2024-09-11T06:05:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55806.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55806.log.gz 2024-09-11T06:05:10.086 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40521.log.gz 2024-09-11T06:05:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56417.log 2024-09-11T06:05:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56541.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56541.log.gz 2024-09-11T06:05:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47318.log 2024-09-11T06:05:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56417.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56417.log.gz 2024-09-11T06:05:10.112 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38489.log 2024-09-11T06:05:10.113 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47318.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47318.log.gz 2024-09-11T06:05:10.116 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44345.log 2024-09-11T06:05:10.117 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38489.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38489.log.gz 2024-09-11T06:05:10.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46786.log 2024-09-11T06:05:10.126 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44345.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44345.log.gz 2024-09-11T06:05:10.136 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38987.log 2024-09-11T06:05:10.144 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46786.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46786.log.gz 2024-09-11T06:05:10.144 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54744.log 2024-09-11T06:05:10.146 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38987.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38987.log.gz 2024-09-11T06:05:10.146 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42337.log 2024-09-11T06:05:10.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54744.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54744.log.gz 2024-09-11T06:05:10.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43321.log 2024-09-11T06:05:10.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42337.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42337.log.gz 2024-09-11T06:05:10.155 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53068.log 2024-09-11T06:05:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43321.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43321.log.gz 2024-09-11T06:05:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33818.log 2024-09-11T06:05:10.172 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38880.log 2024-09-11T06:05:10.173 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33818.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33818.log.gz 2024-09-11T06:05:10.180 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59063.log 2024-09-11T06:05:10.181 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38468.log.gz 2024-09-11T06:05:10.185 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38880.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53068.log.gz 2024-09-11T06:05:10.197 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28714.log 2024-09-11T06:05:10.197 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59063.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59063.log.gz 2024-09-11T06:05:10.210 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38066.log.gz 2024-09-11T06:05:10.211 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64123.log 2024-09-11T06:05:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.28714.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28714.log.gz 2024-09-11T06:05:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30914.log 2024-09-11T06:05:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64123.log.gz 2024-09-11T06:05:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30986.log 2024-09-11T06:05:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.30914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48063.log 2024-09-11T06:05:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.30986.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39708.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34042.log 2024-09-11T06:05:10.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48063.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48063.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34548.log 2024-09-11T06:05:10.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51600.log 2024-09-11T06:05:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34548.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30914.log.gz 2024-09-11T06:05:10.218 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.30986.log.gz 2024-09-11T06:05:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34042.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34548.log.gz 2024-09-11T06:05:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59265.log 2024-09-11T06:05:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51600.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51600.log.gz 2024-09-11T06:05:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37586.log 2024-09-11T06:05:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59265.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59265.log.gz 2024-09-11T06:05:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39586.log 2024-09-11T06:05:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38838.log 2024-09-11T06:05:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39586.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39586.log.gz 2024-09-11T06:05:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43562.log 2024-09-11T06:05:10.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38838.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38838.log.gz 2024-09-11T06:05:10.248 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31420.log 2024-09-11T06:05:10.260 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43562.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43562.log.gz 2024-09-11T06:05:10.260 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56665.log 2024-09-11T06:05:10.261 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31420.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31420.log.gz 2024-09-11T06:05:10.262 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32946.log 2024-09-11T06:05:10.263 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56665.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56665.log.gz 2024-09-11T06:05:10.272 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38548.log.gz 2024-09-11T06:05:10.280 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33430.log 2024-09-11T06:05:10.282 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36997.log 2024-09-11T06:05:10.282 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55186.log 2024-09-11T06:05:10.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62945.log 2024-09-11T06:05:10.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60856.log 2024-09-11T06:05:10.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62466.log 2024-09-11T06:05:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60075.log 2024-09-11T06:05:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34824.log 2024-09-11T06:05:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59018.log 2024-09-11T06:05:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36817.log 2024-09-11T06:05:10.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28639.log 2024-09-11T06:05:10.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47198.log 2024-09-11T06:05:10.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47728.log 2024-09-11T06:05:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64545.log 2024-09-11T06:05:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28921.log 2024-09-11T06:05:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41052.log 2024-09-11T06:05:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64148.log 2024-09-11T06:05:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29320.log 2024-09-11T06:05:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64023.log 2024-09-11T06:05:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38777.log 2024-09-11T06:05:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62354.log 2024-09-11T06:05:10.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39333.log 2024-09-11T06:05:10.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32946.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32946.log.gz 2024-09-11T06:05:10.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33430.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33430.log.gz 2024-09-11T06:05:10.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48196.log 2024-09-11T06:05:10.297 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36997.log.gz 2024-09-11T06:05:10.298 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55186.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55186.log.gz 2024-09-11T06:05:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60856.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60856.log.gz 2024-09-11T06:05:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62466.log: /var/log/ceph/cluster1-client.mirror.60075.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60075.log.gz 2024-09-11T06:05:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34824.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34824.log.gz 2024-09-11T06:05:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59018.log: /var/log/ceph/cluster2-client.mirror.36817.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59018.log.gz 2024-09-11T06:05:10.302 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36817.log.gz 2024-09-11T06:05:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28639.log.gz 2024-09-11T06:05:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47198.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47198.log.gz 2024-09-11T06:05:10.303 INFO:teuthology.orchestra.run.smithi164.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62466.log.gz/var/log/ceph/cluster1-client.mirror.47728.log: 2024-09-11T06:05:10.304 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47728.log.gz 2024-09-11T06:05:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64545.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64545.log.gz 2024-09-11T06:05:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.28921.log: /var/log/ceph/cluster2-client.mirror.41052.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34612.log.gz 2024-09-11T06:05:10.304 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41052.log.gz 2024-09-11T06:05:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62945.log: /var/log/ceph/cluster2-client.mirror.64148.log: /var/log/ceph/cluster1-client.mirror.29320.log: /var/log/ceph/cluster1-client.mirror.64023.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64148.log.gz 2024-09-11T06:05:10.305 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62945.log.gz 2024-09-11T06:05:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38777.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64023.log.gz 2024-09-11T06:05:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62354.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38777.log.gz 2024-09-11T06:05:10.306 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62354.log.gz 2024-09-11T06:05:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39333.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39333.log.gz 2024-09-11T06:05:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50063.log 2024-09-11T06:05:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49943.log 2024-09-11T06:05:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48196.log: /var/log/ceph/cluster1-client.mirror.50063.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50063.log.gz 2024-09-11T06:05:10.308 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48196.log.gz 2024-09-11T06:05:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38365.log 2024-09-11T06:05:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43301.log 2024-09-11T06:05:10.310 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.38365.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49943.log.gz 2024-09-11T06:05:10.310 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36634.log 2024-09-11T06:05:10.310 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38365.log.gz 2024-09-11T06:05:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56479.log 2024-09-11T06:05:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36634.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43301.log.gz 2024-09-11T06:05:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45757.log 2024-09-11T06:05:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56479.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56479.log.gz 2024-09-11T06:05:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59198.log 2024-09-11T06:05:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45757.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45757.log.gz 2024-09-11T06:05:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39888.log 2024-09-11T06:05:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59198.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59198.log.gz 2024-09-11T06:05:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61662.log 2024-09-11T06:05:10.316 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36634.log.gz 2024-09-11T06:05:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39888.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39888.log.gz 2024-09-11T06:05:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37906.log 2024-09-11T06:05:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40320.log 2024-09-11T06:05:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37906.log: 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.28921.log.gz 2024-09-11T06:05:10.337 INFO:teuthology.orchestra.run.smithi164.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29320.log.gz 2024-09-11T06:05:10.348 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56334.log 2024-09-11T06:05:10.349 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40320.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40320.log.gz 2024-09-11T06:05:10.356 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49290.log 2024-09-11T06:05:10.357 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56334.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56334.log.gz 2024-09-11T06:05:10.372 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50668.log 2024-09-11T06:05:10.373 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49290.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49290.log.gz 2024-09-11T06:05:10.380 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43622.log 2024-09-11T06:05:10.381 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50668.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50668.log.gz 2024-09-11T06:05:10.388 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36462.log 2024-09-11T06:05:10.389 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43622.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43622.log.gz 2024-09-11T06:05:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57348.log 2024-09-11T06:05:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36462.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61562.log 2024-09-11T06:05:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57348.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57348.log.gz 2024-09-11T06:05:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57224.log 2024-09-11T06:05:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61562.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61562.log.gz 2024-09-11T06:05:10.416 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36462.log.gz 2024-09-11T06:05:10.432 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29559.log 2024-09-11T06:05:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57224.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57224.log.gz 2024-09-11T06:05:10.443 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39296.log.gz 2024-09-11T06:05:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39046.log 2024-09-11T06:05:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51480.log 2024-09-11T06:05:10.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39046.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29559.log.gz 2024-09-11T06:05:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37285.log 2024-09-11T06:05:10.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51480.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51480.log.gz 2024-09-11T06:05:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49459.log 2024-09-11T06:05:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37285.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37285.log.gz 2024-09-11T06:05:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49642.log 2024-09-11T06:05:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49459.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49459.log.gz 2024-09-11T06:05:10.499 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39375.log.gz 2024-09-11T06:05:10.499 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28282.log 2024-09-11T06:05:10.501 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49642.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49642.log.gz 2024-09-11T06:05:10.501 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63843.log 2024-09-11T06:05:10.502 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28282.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28282.log.gz 2024-09-11T06:05:10.502 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56747.log 2024-09-11T06:05:10.503 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63843.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63843.log.gz 2024-09-11T06:05:10.503 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37666.log 2024-09-11T06:05:10.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56747.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56747.log.gz 2024-09-11T06:05:10.506 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63719.log 2024-09-11T06:05:10.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62376.log 2024-09-11T06:05:10.519 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63719.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63719.log.gz 2024-09-11T06:05:10.527 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36777.log 2024-09-11T06:05:10.549 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39194.log 2024-09-11T06:05:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36777.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61662.log.gz 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36777.log.gz 2024-09-11T06:05:10.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57079.log 2024-09-11T06:05:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39194.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39194.log.gz 2024-09-11T06:05:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46605.log 2024-09-11T06:05:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57079.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57079.log.gz 2024-09-11T06:05:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32161.log 2024-09-11T06:05:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46605.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46605.log.gz 2024-09-11T06:05:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39004.log 2024-09-11T06:05:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.32161.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.32161.log.gz 2024-09-11T06:05:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60008.log 2024-09-11T06:05:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39004.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39004.log.gz 2024-09-11T06:05:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.30273.log 2024-09-11T06:05:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60008.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60008.log.gz 2024-09-11T06:05:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33655.log 2024-09-11T06:05:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.30273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36247.log 2024-09-11T06:05:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33655.log: 82.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33655.log.gz 2024-09-11T06:05:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59333.log 2024-09-11T06:05:10.572 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36247.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38880.log.gz 2024-09-11T06:05:10.576 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36247.log.gz 2024-09-11T06:05:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64363.log 2024-09-11T06:05:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59333.log: 92.6% 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.30273.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.59333.log.gz 2024-09-11T06:05:10.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42798.log 2024-09-11T06:05:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64363.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64363.log.gz 2024-09-11T06:05:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48736.log 2024-09-11T06:05:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42798.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42798.log.gz 2024-09-11T06:05:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39809.log 2024-09-11T06:05:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48736.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48736.log.gz 2024-09-11T06:05:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48395.log 2024-09-11T06:05:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39809.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39809.log.gz 2024-09-11T06:05:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-11T06:05:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48395.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48395.log.gz 2024-09-11T06:05:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34940.log 2024-09-11T06:05:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35320.log 2024-09-11T06:05:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34940.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34940.log.gz 2024-09-11T06:05:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60836.log 2024-09-11T06:05:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35320.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35320.log.gz 2024-09-11T06:05:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62098.log 2024-09-11T06:05:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60836.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60836.log.gz 2024-09-11T06:05:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39686.log 2024-09-11T06:05:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62098.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62098.log.gz 2024-09-11T06:05:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47439.log 2024-09-11T06:05:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39686.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39686.log.gz 2024-09-11T06:05:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49218.log 2024-09-11T06:05:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47439.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47439.log.gz 2024-09-11T06:05:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51904.log 2024-09-11T06:05:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49218.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49218.log.gz 2024-09-11T06:05:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59288.log 2024-09-11T06:05:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51904.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51904.log.gz 2024-09-11T06:05:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36075.log 2024-09-11T06:05:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59288.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59288.log.gz 2024-09-11T06:05:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52975.log 2024-09-11T06:05:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36075.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36075.log.gz 2024-09-11T06:05:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56499.log 2024-09-11T06:05:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52975.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52975.log.gz 2024-09-11T06:05:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61089.log 2024-09-11T06:05:10.610 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-11T06:05:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56499.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56499.log.gz 2024-09-11T06:05:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-11T06:05:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61089.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61089.log.gz 2024-09-11T06:05:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48977.log 2024-09-11T06:05:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2.log: 87.7% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-11T06:05:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.28955.log 2024-09-11T06:05:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48977.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48977.log.gz 2024-09-11T06:05:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39157.log 2024-09-11T06:05:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.28955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46649.log 2024-09-11T06:05:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39157.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39157.log.gz 2024-09-11T06:05:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59783.log 2024-09-11T06:05:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46649.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46649.log.gz 2024-09-11T06:05:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28223.log 2024-09-11T06:05:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59783.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59783.log.gz 2024-09-11T06:05:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31555.log 2024-09-11T06:05:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28223.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28223.log.gz 2024-09-11T06:05:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37926.log 2024-09-11T06:05:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31555.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31555.log.gz 2024-09-11T06:05:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32874.log 2024-09-11T06:05:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37926.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37926.log.gz 2024-09-11T06:05:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36439.log 2024-09-11T06:05:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32874.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32874.log.gz 2024-09-11T06:05:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60489.log 2024-09-11T06:05:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36439.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36439.log.gz 2024-09-11T06:05:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51528.log 2024-09-11T06:05:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60489.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60489.log.gz 2024-09-11T06:05:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52364.log 2024-09-11T06:05:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51528.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51528.log.gz 2024-09-11T06:05:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57740.log 2024-09-11T06:05:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52364.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52364.log.gz 2024-09-11T06:05:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28664.log 2024-09-11T06:05:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57740.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57740.log.gz 2024-09-11T06:05:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50853.log 2024-09-11T06:05:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.28664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28664.log.gz 2024-09-11T06:05:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52907.log 2024-09-11T06:05:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50853.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50853.log.gz 2024-09-11T06:05:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40911.log 2024-09-11T06:05:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52907.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52907.log.gz 2024-09-11T06:05:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61341.log 2024-09-11T06:05:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40911.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40911.log.gz 2024-09-11T06:05:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29448.log 2024-09-11T06:05:10.653 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61341.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61341.log.gz 2024-09-11T06:05:10.653 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59513.log 2024-09-11T06:05:10.654 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.29448.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29448.log.gz 2024-09-11T06:05:10.654 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52686.log 2024-09-11T06:05:10.654 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59513.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59513.log.gz 2024-09-11T06:05:10.655 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31927.log 2024-09-11T06:05:10.656 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52686.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52686.log.gz 2024-09-11T06:05:10.656 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40721.log 2024-09-11T06:05:10.657 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31927.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31927.log.gz 2024-09-11T06:05:10.657 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52710.log 2024-09-11T06:05:10.659 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40721.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40721.log.gz 2024-09-11T06:05:10.659 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62445.log 2024-09-11T06:05:10.660 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52710.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52710.log.gz 2024-09-11T06:05:10.660 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33127.log 2024-09-11T06:05:10.661 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62445.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62445.log.gz 2024-09-11T06:05:10.661 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50608.log 2024-09-11T06:05:10.662 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.33127.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.33127.log.gz 2024-09-11T06:05:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33607.log 2024-09-11T06:05:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50608.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50608.log.gz 2024-09-11T06:05:10.664 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.39397.log 2024-09-11T06:05:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.33607.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.33607.log.gz 2024-09-11T06:05:10.666 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55475.log 2024-09-11T06:05:10.670 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.39397.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.39397.log.gz 2024-09-11T06:05:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50345.log 2024-09-11T06:05:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55475.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55475.log.gz 2024-09-11T06:05:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32683.log 2024-09-11T06:05:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50345.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50345.log.gz 2024-09-11T06:05:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50877.log 2024-09-11T06:05:10.701 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48083.log 2024-09-11T06:05:10.702 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50877.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50877.log.gz 2024-09-11T06:05:10.708 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42943.log 2024-09-11T06:05:10.711 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48083.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48083.log.gz 2024-09-11T06:05:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42943.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42943.log.gz 2024-09-11T06:05:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49797.log 2024-09-11T06:05:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29337.log 2024-09-11T06:05:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49797.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49797.log.gz 2024-09-11T06:05:10.736 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36310.log 2024-09-11T06:05:10.745 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.29337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61154.log 2024-09-11T06:05:10.745 INFO:teuthology.orchestra.run.smithi164.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29337.log.gz 2024-09-11T06:05:10.760 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36310.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62077.log 2024-09-11T06:05:10.761 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36310.log.gz 2024-09-11T06:05:10.761 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61154.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61154.log.gz 2024-09-11T06:05:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50646.log 2024-09-11T06:05:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62077.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62077.log.gz 2024-09-11T06:05:10.792 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39787.log 2024-09-11T06:05:10.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50646.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50646.log.gz 2024-09-11T06:05:10.802 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37666.log.gz 2024-09-11T06:05:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.29006.log 2024-09-11T06:05:10.824 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30729.log 2024-09-11T06:05:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.29006.log: 71.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.29006.log.gz 2024-09-11T06:05:10.836 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30380.log.gz 2024-09-11T06:05:10.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29666.log 2024-09-11T06:05:10.854 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.30729.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.30729.log.gz 2024-09-11T06:05:10.860 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25416.log 2024-09-11T06:05:10.863 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29666.log: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.28955.log.gz 2024-09-11T06:05:10.872 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29666.log.gz 2024-09-11T06:05:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43979.log 2024-09-11T06:05:10.881 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25416.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25416.log.gz 2024-09-11T06:05:10.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33842.log 2024-09-11T06:05:10.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43979.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43979.log.gz 2024-09-11T06:05:10.899 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39046.log.gz 2024-09-11T06:05:10.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53351.log 2024-09-11T06:05:10.913 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33842.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33842.log.gz 2024-09-11T06:05:10.917 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44988.log 2024-09-11T06:05:10.923 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53351.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53351.log.gz 2024-09-11T06:05:10.923 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57327.log 2024-09-11T06:05:10.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44988.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44988.log.gz 2024-09-11T06:05:10.926 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30751.log 2024-09-11T06:05:10.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57327.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57327.log.gz 2024-09-11T06:05:10.927 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43955.log 2024-09-11T06:05:10.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.30751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48678.log 2024-09-11T06:05:10.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43955.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43955.log.gz 2024-09-11T06:05:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38659.log 2024-09-11T06:05:10.930 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48678.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48678.log.gz 2024-09-11T06:05:10.930 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45781.log 2024-09-11T06:05:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38659.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38659.log.gz 2024-09-11T06:05:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54269.log 2024-09-11T06:05:10.941 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34042.log.gz 2024-09-11T06:05:10.941 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.30751.log.gz 2024-09-11T06:05:10.941 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45781.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45781.log.gz 2024-09-11T06:05:10.943 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58613.log 2024-09-11T06:05:10.945 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54269.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54269.log.gz 2024-09-11T06:05:10.946 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29649.log 2024-09-11T06:05:10.946 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36717.log 2024-09-11T06:05:10.946 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58613.log.gz 2024-09-11T06:05:10.946 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.28965.log 2024-09-11T06:05:10.947 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35768.log 2024-09-11T06:05:10.947 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.28965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41151.log 2024-09-11T06:05:10.947 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36717.log.gz 2024-09-11T06:05:10.949 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35768.log: 76.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35768.log.gz 2024-09-11T06:05:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39237.log 2024-09-11T06:05:10.955 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29649.log.gz 2024-09-11T06:05:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41151.log: 94.0% 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41151.log.gz 2024-09-11T06:05:10.956 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37906.log.gz 2024-09-11T06:05:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38719.log 2024-09-11T06:05:10.957 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29303.log 2024-09-11T06:05:10.957 INFO:teuthology.orchestra.run.smithi164.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39237.log.gz/var/log/ceph/cluster2-client.mirror.38719.log: 2024-09-11T06:05:10.957 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40579.log 2024-09-11T06:05:10.958 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.29303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53833.log 2024-09-11T06:05:10.958 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40579.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40579.log.gz 2024-09-11T06:05:10.959 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64273.log 2024-09-11T06:05:10.960 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53833.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37586.log.gz 2024-09-11T06:05:10.961 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53833.log.gz 2024-09-11T06:05:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43907.log 2024-09-11T06:05:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64273.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64273.log.gz 2024-09-11T06:05:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31097.log 2024-09-11T06:05:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43907.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43907.log.gz 2024-09-11T06:05:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43863.log 2024-09-11T06:05:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.28919.log 2024-09-11T06:05:10.972 INFO:teuthology.orchestra.run.smithi164.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29303.log.gz 2024-09-11T06:05:10.972 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31097.log.gz 2024-09-11T06:05:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43863.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43863.log.gz 2024-09-11T06:05:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49843.log 2024-09-11T06:05:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.28919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41484.log 2024-09-11T06:05:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49843.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.28919.log.gz 2024-09-11T06:05:10.979 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49843.log.gz 2024-09-11T06:05:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52628.log 2024-09-11T06:05:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41484.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41484.log.gz 2024-09-11T06:05:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50304.log 2024-09-11T06:05:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37527.log 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52628.log: /var/log/ceph/cluster1-client.mirror.50304.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50304.log.gz 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52628.log.gz 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.28965.log.gz 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41823.log 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50566.log 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37527.log: /var/log/ceph/cluster2-client.mirror.41823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42750.log 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37527.log.gz 2024-09-11T06:05:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45733.log 2024-09-11T06:05:10.986 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42750.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42750.log.gz 2024-09-11T06:05:10.986 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48278.log 2024-09-11T06:05:10.986 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45733.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45733.log.gz 2024-09-11T06:05:10.986 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32826.log 2024-09-11T06:05:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50566.log: /var/log/ceph/cluster1-client.mirror.48278.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48278.log.gz 2024-09-11T06:05:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57983.log 2024-09-11T06:05:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32826.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50566.log.gz 2024-09-11T06:05:10.988 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32826.log.gz 2024-09-11T06:05:10.989 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42774.log 2024-09-11T06:05:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37265.log 2024-09-11T06:05:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57983.log: /var/log/ceph/cluster1-client.mirror.42774.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57983.log.gz 2024-09-11T06:05:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41072.log 2024-09-11T06:05:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37265.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37265.log.gz 2024-09-11T06:05:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28587.log 2024-09-11T06:05:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41072.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41072.log.gz 2024-09-11T06:05:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47419.log 2024-09-11T06:05:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28587.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28587.log.gz 2024-09-11T06:05:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41506.log 2024-09-11T06:05:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47419.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47419.log.gz 2024-09-11T06:05:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35368.log 2024-09-11T06:05:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41506.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42774.log.gz 2024-09-11T06:05:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57960.log 2024-09-11T06:05:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35368.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35368.log.gz 2024-09-11T06:05:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33358.log 2024-09-11T06:05:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57960.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57960.log.gz 2024-09-11T06:05:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36267.log 2024-09-11T06:05:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33358.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33358.log.gz 2024-09-11T06:05:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58185.log 2024-09-11T06:05:11.038 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36267.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44365.log 2024-09-11T06:05:11.040 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58185.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58185.log.gz 2024-09-11T06:05:11.040 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36267.log.gz 2024-09-11T06:05:11.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58073.log 2024-09-11T06:05:11.045 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44365.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44365.log.gz 2024-09-11T06:05:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38284.log 2024-09-11T06:05:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58073.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58073.log.gz 2024-09-11T06:05:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45299.log 2024-09-11T06:05:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38284.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38284.log.gz 2024-09-11T06:05:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28965.log 2024-09-11T06:05:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45299.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45299.log.gz 2024-09-11T06:05:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29686.log 2024-09-11T06:05:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.28965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31735.log 2024-09-11T06:05:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63799.log 2024-09-11T06:05:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.31735.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.31735.log.gz 2024-09-11T06:05:11.072 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28965.log.gz 2024-09-11T06:05:11.073 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29686.log.gz 2024-09-11T06:05:11.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60157.log 2024-09-11T06:05:11.104 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54510.log 2024-09-11T06:05:11.105 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60157.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60157.log.gz 2024-09-11T06:05:11.112 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48234.log 2024-09-11T06:05:11.114 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54510.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54510.log.gz 2024-09-11T06:05:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59873.log 2024-09-11T06:05:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48234.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48234.log.gz 2024-09-11T06:05:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44919.log 2024-09-11T06:05:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59873.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59873.log.gz 2024-09-11T06:05:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50208.log 2024-09-11T06:05:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44919.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44919.log.gz 2024-09-11T06:05:11.149 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63276.log 2024-09-11T06:05:11.149 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50208.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50208.log.gz 2024-09-11T06:05:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53963.log 2024-09-11T06:05:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63276.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63276.log.gz 2024-09-11T06:05:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28197.log 2024-09-11T06:05:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53963.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53963.log.gz 2024-09-11T06:05:11.173 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56355.log 2024-09-11T06:05:11.173 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28197.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28197.log.gz 2024-09-11T06:05:11.180 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47752.log 2024-09-11T06:05:11.181 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56355.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56355.log.gz 2024-09-11T06:05:11.188 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55847.log 2024-09-11T06:05:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47752.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47752.log.gz 2024-09-11T06:05:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55207.log 2024-09-11T06:05:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55847.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55847.log.gz 2024-09-11T06:05:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36937.log 2024-09-11T06:05:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55207.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55207.log.gz 2024-09-11T06:05:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51942.log 2024-09-11T06:05:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36937.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36937.log.gz 2024-09-11T06:05:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40743.log 2024-09-11T06:05:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51942.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51942.log.gz 2024-09-11T06:05:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32415.log 2024-09-11T06:05:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40743.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40743.log.gz 2024-09-11T06:05:11.240 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53047.log 2024-09-11T06:05:11.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32415.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32415.log.gz 2024-09-11T06:05:11.249 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44148.log 2024-09-11T06:05:11.249 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53047.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53047.log.gz 2024-09-11T06:05:11.268 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39787.log.gz 2024-09-11T06:05:11.268 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47118.log 2024-09-11T06:05:11.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44148.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44485.log 2024-09-11T06:05:11.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47118.log: -- replaced with /var/log/ceph/cluster1-client.mirror.44148.log.gz 2024-09-11T06:05:11.279 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47118.log.gz 2024-09-11T06:05:11.279 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49338.log 2024-09-11T06:05:11.281 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44485.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44485.log.gz 2024-09-11T06:05:11.281 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37205.log 2024-09-11T06:05:11.281 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49338.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49338.log.gz 2024-09-11T06:05:11.282 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.62832.log 2024-09-11T06:05:11.282 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37205.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37205.log.gz 2024-09-11T06:05:11.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38247.log 2024-09-11T06:05:11.292 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.62832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61239.log 2024-09-11T06:05:11.293 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38247.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38247.log.gz 2024-09-11T06:05:11.301 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63299.log 2024-09-11T06:05:11.303 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3%/var/log/ceph/cluster2-client.mirror.61239.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61239.log.gz 2024-09-11T06:05:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37120.log 2024-09-11T06:05:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63299.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63299.log.gz 2024-09-11T06:05:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41013.log 2024-09-11T06:05:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.37120.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37120.log.gz 2024-09-11T06:05:11.316 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.62832.log.gz 2024-09-11T06:05:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31788.log 2024-09-11T06:05:11.337 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46143.log 2024-09-11T06:05:11.338 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31788.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.31788.log.gz 2024-09-11T06:05:11.344 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41506.log.gz 2024-09-11T06:05:11.348 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62376.log.gz 2024-09-11T06:05:11.356 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63678.log 2024-09-11T06:05:11.357 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46143.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46143.log.gz 2024-09-11T06:05:11.360 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38719.log.gz 2024-09-11T06:05:11.372 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54050.log 2024-09-11T06:05:11.373 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63678.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63678.log.gz 2024-09-11T06:05:11.374 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53791.log 2024-09-11T06:05:11.375 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54050.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54050.log.gz 2024-09-11T06:05:11.375 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56623.log 2024-09-11T06:05:11.378 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53791.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53791.log.gz 2024-09-11T06:05:11.378 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28839.log 2024-09-11T06:05:11.381 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56623.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56623.log.gz 2024-09-11T06:05:11.381 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49198.log 2024-09-11T06:05:11.382 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28839.log.gz 2024-09-11T06:05:11.382 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32083.log 2024-09-11T06:05:11.383 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49198.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49198.log.gz 2024-09-11T06:05:11.384 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32596.log 2024-09-11T06:05:11.385 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32083.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32083.log.gz 2024-09-11T06:05:11.385 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35005.log 2024-09-11T06:05:11.391 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32596.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32596.log.gz 2024-09-11T06:05:11.392 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42085.log 2024-09-11T06:05:11.397 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35005.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35005.log.gz 2024-09-11T06:05:11.398 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39868.log 2024-09-11T06:05:11.399 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42085.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42085.log.gz 2024-09-11T06:05:11.399 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53240.log 2024-09-11T06:05:11.408 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44827.log 2024-09-11T06:05:11.409 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53240.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53240.log.gz 2024-09-11T06:05:11.424 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55682.log 2024-09-11T06:05:11.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44827.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44827.log.gz 2024-09-11T06:05:11.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58815.log 2024-09-11T06:05:11.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55682.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55682.log.gz 2024-09-11T06:05:11.428 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59400.log 2024-09-11T06:05:11.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58815.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58815.log.gz 2024-09-11T06:05:11.429 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56685.log 2024-09-11T06:05:11.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59400.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59400.log.gz 2024-09-11T06:05:11.430 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49401.log 2024-09-11T06:05:11.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56685.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56685.log.gz 2024-09-11T06:05:11.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52034.log 2024-09-11T06:05:11.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49401.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49401.log.gz 2024-09-11T06:05:11.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60713.log 2024-09-11T06:05:11.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52034.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52034.log.gz 2024-09-11T06:05:11.436 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48808.log 2024-09-11T06:05:11.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60713.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60713.log.gz 2024-09-11T06:05:11.437 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42437.log 2024-09-11T06:05:11.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48808.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48808.log.gz 2024-09-11T06:05:11.438 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34234.log 2024-09-11T06:05:11.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42437.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42437.log.gz 2024-09-11T06:05:11.440 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64311.log 2024-09-11T06:05:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34234.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56251.log 2024-09-11T06:05:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.64311.log: 59.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64311.log.gz 2024-09-11T06:05:11.450 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.34234.log.gz 2024-09-11T06:05:11.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54329.log 2024-09-11T06:05:11.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56251.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56251.log.gz 2024-09-11T06:05:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47901.log 2024-09-11T06:05:11.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54329.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32683.log.gz 2024-09-11T06:05:11.455 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54329.log.gz 2024-09-11T06:05:11.456 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64647.log 2024-09-11T06:05:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47901.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47901.log.gz 2024-09-11T06:05:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45492.log 2024-09-11T06:05:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64647.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64647.log.gz 2024-09-11T06:05:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25264.log 2024-09-11T06:05:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45492.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45492.log.gz 2024-09-11T06:05:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28439.log 2024-09-11T06:05:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25264.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25264.log.gz 2024-09-11T06:05:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35070.log 2024-09-11T06:05:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28439.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28439.log.gz 2024-09-11T06:05:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43473.log 2024-09-11T06:05:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35070.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35070.log.gz 2024-09-11T06:05:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37963.log 2024-09-11T06:05:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43473.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43473.log.gz 2024-09-11T06:05:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32061.log 2024-09-11T06:05:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37963.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37963.log.gz 2024-09-11T06:05:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58208.log 2024-09-11T06:05:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32061.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32061.log.gz 2024-09-11T06:05:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44747.log 2024-09-11T06:05:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58208.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58208.log.gz 2024-09-11T06:05:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44606.log 2024-09-11T06:05:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44747.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44747.log.gz 2024-09-11T06:05:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46721.log 2024-09-11T06:05:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44606.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44606.log.gz 2024-09-11T06:05:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54557.log 2024-09-11T06:05:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46721.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46721.log.gz 2024-09-11T06:05:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55096.log 2024-09-11T06:05:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54557.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54557.log.gz 2024-09-11T06:05:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.61749.log 2024-09-11T06:05:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55096.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55096.log.gz 2024-09-11T06:05:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37057.log 2024-09-11T06:05:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.61749.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.61749.log.gz 2024-09-11T06:05:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46123.log 2024-09-11T06:05:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37057.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37057.log.gz 2024-09-11T06:05:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30273.log 2024-09-11T06:05:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46123.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46123.log.gz 2024-09-11T06:05:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36224.log 2024-09-11T06:05:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.30273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57699.log 2024-09-11T06:05:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36224.log: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.30273.log.gz 2024-09-11T06:05:11.494 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36224.log.gz 2024-09-11T06:05:11.494 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54028.log 2024-09-11T06:05:11.495 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57699.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57699.log.gz 2024-09-11T06:05:11.496 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63657.log 2024-09-11T06:05:11.498 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54028.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54028.log.gz 2024-09-11T06:05:11.498 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36353.log 2024-09-11T06:05:11.499 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63657.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63657.log.gz 2024-09-11T06:05:11.499 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53409.log 2024-09-11T06:05:11.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36353.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36353.log.gz 2024-09-11T06:05:11.506 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46846.log 2024-09-11T06:05:11.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53409.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53409.log.gz 2024-09-11T06:05:11.507 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48415.log 2024-09-11T06:05:11.508 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46846.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46846.log.gz 2024-09-11T06:05:11.509 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39254.log 2024-09-11T06:05:11.511 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48415.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48415.log.gz 2024-09-11T06:05:11.511 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38105.log 2024-09-11T06:05:11.513 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39254.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39254.log.gz 2024-09-11T06:05:11.513 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49314.log 2024-09-11T06:05:11.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38105.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38105.log.gz 2024-09-11T06:05:11.515 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35680.log 2024-09-11T06:05:11.515 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49314.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49314.log.gz 2024-09-11T06:05:11.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40004.log 2024-09-11T06:05:11.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35680.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35680.log.gz 2024-09-11T06:05:11.517 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45902.log 2024-09-11T06:05:11.518 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40004.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40004.log.gz 2024-09-11T06:05:11.518 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29204.log 2024-09-11T06:05:11.519 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45902.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45902.log.gz 2024-09-11T06:05:11.520 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49507.log 2024-09-11T06:05:11.520 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.29204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29204.log.gz 2024-09-11T06:05:11.521 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30894.log 2024-09-11T06:05:11.521 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49507.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49507.log.gz 2024-09-11T06:05:11.522 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32483.log 2024-09-11T06:05:11.523 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.30894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58478.log 2024-09-11T06:05:11.523 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32483.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32483.log.gz 2024-09-11T06:05:11.524 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63399.log 2024-09-11T06:05:11.525 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30894.log.gz 2024-09-11T06:05:11.525 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58478.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58478.log.gz 2024-09-11T06:05:11.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52666.log 2024-09-11T06:05:11.526 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63399.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63399.log.gz 2024-09-11T06:05:11.527 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57849.log 2024-09-11T06:05:11.528 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52666.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52666.log.gz 2024-09-11T06:05:11.528 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44024.log 2024-09-11T06:05:11.529 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57849.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57849.log.gz 2024-09-11T06:05:11.529 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63885.log 2024-09-11T06:05:11.531 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44024.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41823.log.gz 2024-09-11T06:05:11.531 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44024.log.gz 2024-09-11T06:05:11.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38426.log 2024-09-11T06:05:11.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42379.log 2024-09-11T06:05:11.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63885.log: /var/log/ceph/cluster2-client.mirror.38426.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63885.log.gz 2024-09-11T06:05:11.533 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64467.log 2024-09-11T06:05:11.534 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42379.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38426.log.gz 2024-09-11T06:05:11.534 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42379.log.gz 2024-09-11T06:05:11.534 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52586.log 2024-09-11T06:05:11.535 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32016.log 2024-09-11T06:05:11.535 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64467.log.gz 2024-09-11T06:05:11.535 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56210.log 2024-09-11T06:05:11.537 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32016.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52586.log.gz 2024-09-11T06:05:11.540 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37078.log 2024-09-11T06:05:11.541 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56210.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32016.log.gz 2024-09-11T06:05:11.541 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56210.log.gz 2024-09-11T06:05:11.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52782.log 2024-09-11T06:05:11.542 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37017.log 2024-09-11T06:05:11.543 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52782.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37078.log.gz 2024-09-11T06:05:11.543 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52782.log.gz 2024-09-11T06:05:11.543 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48519.log 2024-09-11T06:05:11.544 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29593.log 2024-09-11T06:05:11.544 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48519.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37017.log.gz 2024-09-11T06:05:11.544 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48519.log.gz 2024-09-11T06:05:11.545 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57492.log 2024-09-11T06:05:11.545 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64389.log 2024-09-11T06:05:11.545 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29593.log: /var/log/ceph/cluster2-client.mirror.57492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51139.log 2024-09-11T06:05:11.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64389.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64389.log.gz 2024-09-11T06:05:11.546 INFO:teuthology.orchestra.run.smithi164.stderr: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.29593.log.gz 2024-09-11T06:05:11.546 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34484.log 2024-09-11T06:05:11.548 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51139.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57492.log.gz 2024-09-11T06:05:11.548 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51139.log.gz 2024-09-11T06:05:11.549 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62766.log 2024-09-11T06:05:11.549 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38184.log 2024-09-11T06:05:11.550 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34484.log: /var/log/ceph/cluster1-client.mirror.62766.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34484.log.gz 2024-09-11T06:05:11.550 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62766.log.gz 2024-09-11T06:05:11.550 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55869.log 2024-09-11T06:05:11.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38184.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38184.log.gz 2024-09-11T06:05:11.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29354.log 2024-09-11T06:05:11.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50521.log 2024-09-11T06:05:11.552 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55869.log: /var/log/ceph/cluster1-client.mirror.29354.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55869.log.gz 2024-09-11T06:05:11.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50256.log 2024-09-11T06:05:11.553 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46697.log 2024-09-11T06:05:11.554 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50521.log: /var/log/ceph/cluster1-client.mirror.50256.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50521.log.gz 2024-09-11T06:05:11.554 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50256.log.gz 2024-09-11T06:05:11.554 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42292.log 2024-09-11T06:05:11.555 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36568.log 2024-09-11T06:05:11.555 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46697.log: /var/log/ceph/cluster1-client.mirror.42292.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46697.log.gz 2024-09-11T06:05:11.555 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42292.log.gz 2024-09-11T06:05:11.556 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55785.log 2024-09-11T06:05:11.556 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64073.log 2024-09-11T06:05:11.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36568.log: /var/log/ceph/cluster2-client.mirror.55785.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55785.log.gz 2024-09-11T06:05:11.560 INFO:teuthology.orchestra.run.smithi164.stderr: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.29354.log.gz 2024-09-11T06:05:11.561 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36568.log.gz 2024-09-11T06:05:11.561 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39545.log 2024-09-11T06:05:11.562 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36591.log 2024-09-11T06:05:11.562 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64073.log: /var/log/ceph/cluster2-client.mirror.39545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64073.log.gz 2024-09-11T06:05:11.563 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.45030.log 2024-09-11T06:05:11.569 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36591.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36591.log.gz 2024-09-11T06:05:11.569 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36161.log 2024-09-11T06:05:11.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.45030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35392.log 2024-09-11T06:05:11.591 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36161.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36161.log.gz 2024-09-11T06:05:11.593 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.45030.log.gz 2024-09-11T06:05:11.593 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51022.log 2024-09-11T06:05:11.594 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35392.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35392.log.gz 2024-09-11T06:05:11.594 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43803.log 2024-09-11T06:05:11.595 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51022.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51022.log.gz 2024-09-11T06:05:11.595 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32459.log 2024-09-11T06:05:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43803.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43803.log.gz 2024-09-11T06:05:11.598 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.34066.log 2024-09-11T06:05:11.599 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32459.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32459.log.gz 2024-09-11T06:05:11.599 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61640.log 2024-09-11T06:05:11.605 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.34066.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.34066.log.gz 2024-09-11T06:05:11.606 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40499.log 2024-09-11T06:05:11.606 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61640.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61640.log.gz 2024-09-11T06:05:11.607 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48326.log 2024-09-11T06:05:11.608 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40499.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40499.log.gz 2024-09-11T06:05:11.608 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38226.log 2024-09-11T06:05:11.609 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48326.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48326.log.gz 2024-09-11T06:05:11.609 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45581.log 2024-09-11T06:05:11.624 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38328.log 2024-09-11T06:05:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45581.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45581.log.gz 2024-09-11T06:05:11.640 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52103.log 2024-09-11T06:05:11.641 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38328.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38328.log.gz 2024-09-11T06:05:11.648 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39967.log 2024-09-11T06:05:11.650 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52103.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52103.log.gz 2024-09-11T06:05:11.664 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34436.log 2024-09-11T06:05:11.665 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39967.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39967.log.gz 2024-09-11T06:05:11.672 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46625.log 2024-09-11T06:05:11.673 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34436.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34436.log.gz 2024-09-11T06:05:11.681 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.61407.log 2024-09-11T06:05:11.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46625.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46625.log.gz 2024-09-11T06:05:11.685 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-11T06:05:11.685 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55269.log 2024-09-11T06:05:11.689 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.61407.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.61407.log.gz 2024-09-11T06:05:11.690 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28764.log 2024-09-11T06:05:11.690 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55269.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55269.log.gz 2024-09-11T06:05:11.691 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40623.log 2024-09-11T06:05:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.28764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28764.log.gz 2024-09-11T06:05:11.692 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33978.log 2024-09-11T06:05:11.693 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40623.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40623.log.gz 2024-09-11T06:05:11.693 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62004.log 2024-09-11T06:05:11.698 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33978.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33978.log.gz 2024-09-11T06:05:11.699 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37688.log 2024-09-11T06:05:11.713 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25486.log 2024-09-11T06:05:11.714 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37688.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37688.log.gz 2024-09-11T06:05:11.714 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39846.log 2024-09-11T06:05:11.714 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25486.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25486.log.gz 2024-09-11T06:05:11.717 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31119.log.gz 2024-09-11T06:05:11.721 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45340.log 2024-09-11T06:05:11.721 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39846.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39846.log.gz 2024-09-11T06:05:11.722 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41094.log 2024-09-11T06:05:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45340.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45340.log.gz 2024-09-11T06:05:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44244.log 2024-09-11T06:05:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41094.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39868.log.gz 2024-09-11T06:05:11.732 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63799.log.gz 2024-09-11T06:05:11.732 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54088.log 2024-09-11T06:05:11.733 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37644.log 2024-09-11T06:05:11.733 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54088.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44244.log.gz 2024-09-11T06:05:11.734 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37785.log 2024-09-11T06:05:11.735 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54088.log.gz 2024-09-11T06:05:11.735 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39925.log 2024-09-11T06:05:11.735 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37644.log.gz 2024-09-11T06:05:11.736 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40559.log 2024-09-11T06:05:11.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37785.log: /var/log/ceph/cluster2-client.mirror.39925.log: 87.6% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37785.log.gz 2024-09-11T06:05:11.736 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39925.log.gz 2024-09-11T06:05:11.736 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28889.log 2024-09-11T06:05:11.737 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57100.log 2024-09-11T06:05:11.739 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28889.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28889.log.gz 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40559.log.gz 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43256.log 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58230.log 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43256.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57100.log.gz 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43256.log.gz 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51159.log 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41543.log 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58230.log.gz 2024-09-11T06:05:11.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43140.log 2024-09-11T06:05:11.741 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41543.log: 92.2% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41543.log.gz 2024-09-11T06:05:11.741 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51159.log.gz 2024-09-11T06:05:11.742 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61620.log 2024-09-11T06:05:11.742 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36290.log 2024-09-11T06:05:11.743 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61620.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43140.log.gz 2024-09-11T06:05:11.744 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61620.log.gz 2024-09-11T06:05:11.744 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43184.log 2024-09-11T06:05:11.744 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54992.log 2024-09-11T06:05:11.744 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43184.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43184.log.gz 2024-09-11T06:05:11.745 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35946.log 2024-09-11T06:05:11.746 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54992.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54992.log.gz 2024-09-11T06:05:11.746 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34916.log 2024-09-11T06:05:11.749 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35946.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36290.log.gz 2024-09-11T06:05:11.751 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35946.log.gz 2024-09-11T06:05:11.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42417.log 2024-09-11T06:05:11.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47622.log 2024-09-11T06:05:11.754 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34916.log: /var/log/ceph/cluster2-client.mirror.42417.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34916.log.gz 2024-09-11T06:05:11.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33794.log 2024-09-11T06:05:11.755 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42417.log.gz 2024-09-11T06:05:11.758 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47622.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47622.log.gz 2024-09-11T06:05:11.758 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58365.log 2024-09-11T06:05:11.758 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52927.log 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33794.log: /var/log/ceph/cluster1-client.mirror.58365.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58365.log.gz 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33794.log.gz 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58950.log 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40275.log 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52927.log: /var/log/ceph/cluster1-client.mirror.58950.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52927.log.gz 2024-09-11T06:05:11.759 INFO:teuthology.orchestra.run.smithi164.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58950.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58860.log 2024-09-11T06:05:11.760 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:11.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41131.log 2024-09-11T06:05:11.773 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58860.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58860.log.gz 2024-09-11T06:05:11.774 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38858.log 2024-09-11T06:05:11.775 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41131.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41131.log.gz 2024-09-11T06:05:11.776 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49160.log 2024-09-11T06:05:11.776 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38858.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38858.log.gz 2024-09-11T06:05:11.777 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56272.log 2024-09-11T06:05:11.778 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49160.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49160.log.gz 2024-09-11T06:05:11.778 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43497.log 2024-09-11T06:05:11.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56272.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56272.log.gz 2024-09-11T06:05:11.780 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57410.log 2024-09-11T06:05:11.781 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28516.log 2024-09-11T06:05:11.781 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43497.log.gz/var/log/ceph/cluster2-client.mirror.57410.log: 2024-09-11T06:05:11.781 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41212.log 2024-09-11T06:05:11.782 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28516.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28516.log.gz 2024-09-11T06:05:11.782 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46745.log 2024-09-11T06:05:11.783 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41212.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41212.log.gz 2024-09-11T06:05:11.783 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46063.log 2024-09-11T06:05:11.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46745.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46745.log.gz 2024-09-11T06:05:11.785 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63864.log 2024-09-11T06:05:11.787 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46063.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46063.log.gz 2024-09-11T06:05:11.787 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54682.log 2024-09-11T06:05:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63864.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63864.log.gz 2024-09-11T06:05:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47246.log 2024-09-11T06:05:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54682.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54682.log.gz 2024-09-11T06:05:11.791 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48254.log 2024-09-11T06:05:11.792 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47246.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47246.log.gz 2024-09-11T06:05:11.793 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55517.log 2024-09-11T06:05:11.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48254.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48254.log.gz 2024-09-11T06:05:11.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28613.log 2024-09-11T06:05:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55517.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55517.log.gz 2024-09-11T06:05:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49266.log 2024-09-11T06:05:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28613.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28613.log.gz 2024-09-11T06:05:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33998.log 2024-09-11T06:05:11.799 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49266.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57410.log.gz 2024-09-11T06:05:11.799 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49266.log.gz 2024-09-11T06:05:11.799 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35167.log 2024-09-11T06:05:11.803 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33998.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33998.log.gz 2024-09-11T06:05:11.804 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39216.log 2024-09-11T06:05:11.805 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35167.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35167.log.gz 2024-09-11T06:05:11.806 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64193.log 2024-09-11T06:05:11.820 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58658.log 2024-09-11T06:05:11.821 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64193.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64193.log.gz 2024-09-11T06:05:11.834 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39545.log.gz 2024-09-11T06:05:11.844 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41447.log 2024-09-11T06:05:11.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58658.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58658.log.gz 2024-09-11T06:05:11.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51576.log 2024-09-11T06:05:11.847 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41447.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41447.log.gz 2024-09-11T06:05:11.847 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.50668.log 2024-09-11T06:05:11.848 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51576.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51576.log.gz 2024-09-11T06:05:11.848 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58590.log 2024-09-11T06:05:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.50668.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41013.log.gz 2024-09-11T06:05:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.33175.log 2024-09-11T06:05:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58590.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58590.log.gz 2024-09-11T06:05:11.858 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62601.log 2024-09-11T06:05:11.860 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.33175.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.33175.log.gz 2024-09-11T06:05:11.860 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28689.log 2024-09-11T06:05:11.861 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62601.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62601.log.gz 2024-09-11T06:05:11.861 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41464.log 2024-09-11T06:05:11.862 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.50668.log.gz/var/log/ceph/cluster1-client.admin.28689.log: 2024-09-11T06:05:11.862 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28689.log.gz 2024-09-11T06:05:11.862 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56107.log 2024-09-11T06:05:11.863 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41781.log 2024-09-11T06:05:11.863 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41464.log: /var/log/ceph/cluster2-client.mirror.56107.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56107.log.gz 2024-09-11T06:05:11.864 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41464.log.gz 2024-09-11T06:05:11.864 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48543.log 2024-09-11T06:05:11.865 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41781.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41781.log.gz 2024-09-11T06:05:11.865 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60053.log 2024-09-11T06:05:11.867 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46545.log 2024-09-11T06:05:11.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48543.log: /var/log/ceph/cluster1-client.mirror.60053.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60053.log.gz 2024-09-11T06:05:11.867 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48543.log.gz 2024-09-11T06:05:11.867 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39984.log 2024-09-11T06:05:11.868 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49001.log 2024-09-11T06:05:11.868 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46545.log: /var/log/ceph/cluster2-client.mirror.39984.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39984.log.gz 2024-09-11T06:05:11.869 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46545.log.gz 2024-09-11T06:05:11.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46043.log 2024-09-11T06:05:11.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64597.log 2024-09-11T06:05:11.871 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49001.log: /var/log/ceph/cluster2-client.mirror.46043.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49001.log.gz 2024-09-11T06:05:11.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40601.log 2024-09-11T06:05:11.874 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64597.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64597.log.gz 2024-09-11T06:05:11.874 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46043.log.gz 2024-09-11T06:05:11.874 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58703.log 2024-09-11T06:05:11.874 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55703.log 2024-09-11T06:05:11.874 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40601.log: /var/log/ceph/cluster1-client.mirror.58703.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58703.log.gz 2024-09-11T06:05:11.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47098.log 2024-09-11T06:05:11.889 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48350.log 2024-09-11T06:05:11.890 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55703.log: /var/log/ceph/cluster2-client.mirror.47098.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55703.log.gz 2024-09-11T06:05:11.890 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40542.log 2024-09-11T06:05:11.891 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48350.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47098.log.gz 2024-09-11T06:05:11.891 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48350.log.gz 2024-09-11T06:05:11.891 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56042.log 2024-09-11T06:05:11.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51239.log 2024-09-11T06:05:11.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40542.log: /var/log/ceph/cluster2-client.mirror.56042.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40542.log.gz 2024-09-11T06:05:11.893 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56042.log.gz 2024-09-11T06:05:11.893 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51422.log 2024-09-11T06:05:11.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51239.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51239.log.gz 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25381.log 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51422.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51422.log.gz 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41722.log 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25381.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25381.log.gz 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51641.log 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41722.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41722.log.gz 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.32209.log 2024-09-11T06:05:11.900 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51641.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51641.log.gz 2024-09-11T06:05:11.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45516.log 2024-09-11T06:05:11.905 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.32209.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.32209.log.gz 2024-09-11T06:05:11.905 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49600.log 2024-09-11T06:05:11.907 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45516.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45516.log.gz 2024-09-11T06:05:11.907 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35704.log 2024-09-11T06:05:11.908 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49600.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49600.log.gz 2024-09-11T06:05:11.908 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53592.log 2024-09-11T06:05:11.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35704.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35704.log.gz 2024-09-11T06:05:11.910 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61603.log 2024-09-11T06:05:11.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53592.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53592.log.gz 2024-09-11T06:05:11.911 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59985.log 2024-09-11T06:05:11.912 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61603.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61603.log.gz 2024-09-11T06:05:11.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39068.log 2024-09-11T06:05:11.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59985.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59985.log.gz 2024-09-11T06:05:11.914 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47680.log 2024-09-11T06:05:11.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39068.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39068.log.gz 2024-09-11T06:05:11.915 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40891.log 2024-09-11T06:05:11.916 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47680.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47680.log.gz 2024-09-11T06:05:11.916 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54888.log 2024-09-11T06:05:11.918 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40891.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40891.log.gz 2024-09-11T06:05:11.918 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40954.log 2024-09-11T06:05:11.921 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54888.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54888.log.gz 2024-09-11T06:05:11.921 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34344.log 2024-09-11T06:05:11.922 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40954.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40954.log.gz 2024-09-11T06:05:11.922 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58253.log 2024-09-11T06:05:11.924 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34344.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34344.log.gz 2024-09-11T06:05:11.924 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41334.log 2024-09-11T06:05:11.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58253.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58253.log.gz 2024-09-11T06:05:11.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55033.log 2024-09-11T06:05:11.926 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41334.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41334.log.gz 2024-09-11T06:05:11.926 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36797.log 2024-09-11T06:05:11.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55033.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55033.log.gz 2024-09-11T06:05:11.927 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59580.log 2024-09-11T06:05:11.929 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36797.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36797.log.gz 2024-09-11T06:05:11.929 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59895.log 2024-09-11T06:05:11.930 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59580.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59580.log.gz 2024-09-11T06:05:11.930 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41744.log 2024-09-11T06:05:11.931 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59895.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59895.log.gz 2024-09-11T06:05:11.931 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58995.log 2024-09-11T06:05:11.950 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56169.log 2024-09-11T06:05:11.950 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58995.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58995.log.gz 2024-09-11T06:05:11.964 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54008.log 2024-09-11T06:05:11.965 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56169.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56169.log.gz 2024-09-11T06:05:11.972 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61198.log 2024-09-11T06:05:11.974 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54008.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54008.log.gz 2024-09-11T06:05:11.980 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46384.log 2024-09-11T06:05:11.981 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61198.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61198.log.gz 2024-09-11T06:05:11.988 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58973.log 2024-09-11T06:05:11.989 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46384.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46384.log.gz 2024-09-11T06:05:11.989 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41094.log.gz 2024-09-11T06:05:11.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46567.log 2024-09-11T06:05:11.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58973.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58973.log.gz 2024-09-11T06:05:11.991 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32806.log 2024-09-11T06:05:11.992 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46567.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46567.log.gz 2024-09-11T06:05:11.993 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40164.log 2024-09-11T06:05:11.994 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32806.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32806.log.gz 2024-09-11T06:05:11.994 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45950.log 2024-09-11T06:05:11.996 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.40164.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40164.log.gz 2024-09-11T06:05:11.997 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31667.log 2024-09-11T06:05:11.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45950.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45950.log.gz 2024-09-11T06:05:11.997 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54373.log 2024-09-11T06:05:12.003 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31667.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31667.log.gz 2024-09-11T06:05:12.004 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58140.log 2024-09-11T06:05:12.004 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54373.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54373.log.gz 2024-09-11T06:05:12.005 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58298.log 2024-09-11T06:05:12.005 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58140.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58140.log.gz 2024-09-11T06:05:12.006 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46964.log 2024-09-11T06:05:12.006 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58298.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58298.log.gz 2024-09-11T06:05:12.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61497.log 2024-09-11T06:05:12.007 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46964.log: 82.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46964.log.gz 2024-09-11T06:05:12.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59175.log 2024-09-11T06:05:12.008 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61497.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61497.log.gz 2024-09-11T06:05:12.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63613.log 2024-09-11T06:05:12.009 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59175.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59175.log.gz 2024-09-11T06:05:12.009 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54132.log 2024-09-11T06:05:12.011 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63613.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63613.log.gz 2024-09-11T06:05:12.012 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62054.log 2024-09-11T06:05:12.012 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54132.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54132.log.gz 2024-09-11T06:05:12.013 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28353.log 2024-09-11T06:05:12.013 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62054.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62054.log.gz 2024-09-11T06:05:12.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46456.log 2024-09-11T06:05:12.014 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28353.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28353.log.gz 2024-09-11T06:05:12.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43931.log 2024-09-11T06:05:12.015 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46456.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46456.log.gz 2024-09-11T06:05:12.015 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42357.log 2024-09-11T06:05:12.016 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43931.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43931.log.gz 2024-09-11T06:05:12.016 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47945.log 2024-09-11T06:05:12.018 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42357.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42357.log.gz 2024-09-11T06:05:12.019 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40640.log 2024-09-11T06:05:12.019 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47945.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47945.log.gz 2024-09-11T06:05:12.020 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52827.log 2024-09-11T06:05:12.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40640.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40640.log.gz 2024-09-11T06:05:12.021 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32187.log 2024-09-11T06:05:12.023 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52827.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52827.log.gz 2024-09-11T06:05:12.024 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55723.log 2024-09-11T06:05:12.025 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32187.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32187.log.gz 2024-09-11T06:05:12.025 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59423.log 2024-09-11T06:05:12.029 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55723.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55723.log.gz 2024-09-11T06:05:12.029 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38226.log.gz 2024-09-11T06:05:12.029 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29474.log 2024-09-11T06:05:12.029 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59423.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59423.log.gz 2024-09-11T06:05:12.029 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38626.log 2024-09-11T06:05:12.030 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.29474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.46964.log 2024-09-11T06:05:12.030 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47535.log 2024-09-11T06:05:12.031 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38626.log: /var/log/ceph/cluster1-client.mirror.3.46964.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38626.log.gz 2024-09-11T06:05:12.031 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44943.log 2024-09-11T06:05:12.031 INFO:teuthology.orchestra.run.smithi164.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.29474.log.gz 2024-09-11T06:05:12.032 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47535.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47535.log.gz 2024-09-11T06:05:12.032 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30962.log 2024-09-11T06:05:12.033 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44943.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44943.log.gz 2024-09-11T06:05:12.033 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36138.log 2024-09-11T06:05:12.034 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.30962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55599.log 2024-09-11T06:05:12.035 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.46964.log.gz 2024-09-11T06:05:12.035 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30962.log.gz 2024-09-11T06:05:12.040 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36138.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36138.log.gz 2024-09-11T06:05:12.041 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36757.log 2024-09-11T06:05:12.041 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50998.log 2024-09-11T06:05:12.042 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36757.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36757.log.gz 2024-09-11T06:05:12.043 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34505.log 2024-09-11T06:05:12.044 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50998.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50998.log.gz 2024-09-11T06:05:12.044 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55599.log.gz 2024-09-11T06:05:12.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42839.log 2024-09-11T06:05:12.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.34234.log 2024-09-11T06:05:12.046 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34505.log: /var/log/ceph/cluster1-client.mirror.42839.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42839.log.gz 2024-09-11T06:05:12.049 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34505.log.gz 2024-09-11T06:05:12.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37768.log 2024-09-11T06:05:12.050 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.34234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60194.log 2024-09-11T06:05:12.051 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37768.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37768.log.gz 2024-09-11T06:05:12.051 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.34710.log 2024-09-11T06:05:12.052 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60194.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60194.log.gz 2024-09-11T06:05:12.052 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38007.log 2024-09-11T06:05:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.34710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48042.log 2024-09-11T06:05:12.065 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38007.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38007.log.gz 2024-09-11T06:05:12.073 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.34710.log.gz 2024-09-11T06:05:12.076 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.63476.log 2024-09-11T06:05:12.077 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48042.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48042.log.gz 2024-09-11T06:05:12.078 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51701.log 2024-09-11T06:05:12.080 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.63476.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.63476.log.gz 2024-09-11T06:05:12.081 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49555.log 2024-09-11T06:05:12.082 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51701.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51701.log.gz 2024-09-11T06:05:12.083 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40125.log 2024-09-11T06:05:12.083 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49555.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49555.log.gz 2024-09-11T06:05:12.084 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42047.log 2024-09-11T06:05:12.084 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40125.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40125.log.gz 2024-09-11T06:05:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54536.log 2024-09-11T06:05:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42047.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42047.log.gz 2024-09-11T06:05:12.087 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29025.log 2024-09-11T06:05:12.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54536.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54536.log.gz 2024-09-11T06:05:12.088 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52847.log 2024-09-11T06:05:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.29025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54702.log 2024-09-11T06:05:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52847.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29025.log.gz 2024-09-11T06:05:12.091 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52847.log.gz 2024-09-11T06:05:12.092 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48374.log 2024-09-11T06:05:12.095 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54702.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54702.log.gz 2024-09-11T06:05:12.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48302.log 2024-09-11T06:05:12.096 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48374.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48374.log.gz 2024-09-11T06:05:12.096 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28127.log 2024-09-11T06:05:12.097 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48302.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48302.log.gz 2024-09-11T06:05:12.097 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43343.log 2024-09-11T06:05:12.098 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28127.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28127.log.gz 2024-09-11T06:05:12.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60465.log 2024-09-11T06:05:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43343.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43343.log.gz 2024-09-11T06:05:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38264.log 2024-09-11T06:05:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60465.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60465.log.gz 2024-09-11T06:05:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49483.log 2024-09-11T06:05:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38264.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38264.log.gz 2024-09-11T06:05:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31189.log 2024-09-11T06:05:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49483.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49483.log.gz 2024-09-11T06:05:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39826.log 2024-09-11T06:05:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.31189.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.31189.log.gz 2024-09-11T06:05:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45561.log 2024-09-11T06:05:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39826.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39826.log.gz 2024-09-11T06:05:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53329.log 2024-09-11T06:05:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45561.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45561.log.gz 2024-09-11T06:05:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63341.log 2024-09-11T06:05:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53329.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.2.34234.log.gz 2024-09-11T06:05:12.112 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53329.log.gz 2024-09-11T06:05:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62966.log 2024-09-11T06:05:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38697.log 2024-09-11T06:05:12.114 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63341.log: /var/log/ceph/cluster2-client.mirror.62966.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63341.log.gz 2024-09-11T06:05:12.114 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62966.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43102.log 2024-09-11T06:05:12.114 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.114 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38697.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38697.log.gz 2024-09-11T06:05:12.115 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62255.log 2024-09-11T06:05:12.115 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51118.log 2024-09-11T06:05:12.116 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43102.log: /var/log/ceph/cluster2-client.mirror.62255.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62255.log.gz 2024-09-11T06:05:12.116 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43102.log.gz 2024-09-11T06:05:12.116 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29405.log 2024-09-11T06:05:12.117 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49097.log 2024-09-11T06:05:12.117 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51118.log: /var/log/ceph/cluster1-client.mirror.29405.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51118.log.gz 2024-09-11T06:05:12.118 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53871.log 2024-09-11T06:05:12.118 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53529.log 2024-09-11T06:05:12.119 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49097.log: /var/log/ceph/cluster2-client.mirror.53871.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.29405.log.gz 2024-09-11T06:05:12.119 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49097.log.gz 2024-09-11T06:05:12.120 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61133.log 2024-09-11T06:05:12.120 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53529.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53871.log.gz 2024-09-11T06:05:12.120 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53529.log.gz 2024-09-11T06:05:12.121 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60139.log 2024-09-11T06:05:12.121 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58050.log 2024-09-11T06:05:12.122 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61133.log: /var/log/ceph/cluster2-client.mirror.60139.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60139.log.gz 2024-09-11T06:05:12.122 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61133.log.gz 2024-09-11T06:05:12.123 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57162.log 2024-09-11T06:05:12.123 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45275.log 2024-09-11T06:05:12.124 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58050.log: /var/log/ceph/cluster2-client.mirror.57162.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58050.log.gz 2024-09-11T06:05:12.124 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57162.log.gz 2024-09-11T06:05:12.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57265.log 2024-09-11T06:05:12.125 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45275.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45275.log.gz 2024-09-11T06:05:12.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43542.log 2024-09-11T06:05:12.126 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43762.log 2024-09-11T06:05:12.126 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57265.log: /var/log/ceph/cluster2-client.mirror.43542.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57265.log.gz 2024-09-11T06:05:12.127 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39649.log 2024-09-11T06:05:12.128 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43762.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43542.log.gz 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43762.log.gz 2024-09-11T06:05:12.128 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.128 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58320.log 2024-09-11T06:05:12.128 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48784.log 2024-09-11T06:05:12.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39649.log: 80.4%/var/log/ceph/cluster1-client.mirror.58320.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39649.log.gz 2024-09-11T06:05:12.129 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41427.log 2024-09-11T06:05:12.130 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58320.log.gz 2024-09-11T06:05:12.130 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48784.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48784.log.gz 2024-09-11T06:05:12.130 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61297.log 2024-09-11T06:05:12.130 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.62120.log 2024-09-11T06:05:12.131 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41427.log: /var/log/ceph/cluster2-client.mirror.61297.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61297.log.gz 2024-09-11T06:05:12.140 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35656.log 2024-09-11T06:05:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.62120.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.62120.log.gz 2024-09-11T06:05:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63820.log 2024-09-11T06:05:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35656.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35656.log.gz 2024-09-11T06:05:12.148 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38900.log 2024-09-11T06:05:12.149 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63820.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63820.log.gz 2024-09-11T06:05:12.149 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40083.log 2024-09-11T06:05:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38900.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38900.log.gz 2024-09-11T06:05:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40789.log 2024-09-11T06:05:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40083.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40083.log.gz 2024-09-11T06:05:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53309.log 2024-09-11T06:05:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40789.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40789.log.gz 2024-09-11T06:05:12.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40442.log 2024-09-11T06:05:12.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53309.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53309.log.gz 2024-09-11T06:05:12.155 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45709.log 2024-09-11T06:05:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55980.log 2024-09-11T06:05:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45709.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45709.log.gz 2024-09-11T06:05:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35048.log 2024-09-11T06:05:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55980.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55980.log.gz 2024-09-11T06:05:12.191 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40275.log.gz 2024-09-11T06:05:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38409.log 2024-09-11T06:05:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35048.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35048.log.gz 2024-09-11T06:05:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53770.log 2024-09-11T06:05:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38409.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38409.log.gz 2024-09-11T06:05:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63320.log 2024-09-11T06:05:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53770.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53770.log.gz 2024-09-11T06:05:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60878.log 2024-09-11T06:05:12.201 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63320.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63320.log.gz 2024-09-11T06:05:12.201 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42578.log 2024-09-11T06:05:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60878.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41744.log.gz 2024-09-11T06:05:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42678.log 2024-09-11T06:05:12.209 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42578.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42578.log.gz 2024-09-11T06:05:12.212 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55827.log 2024-09-11T06:05:12.213 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42678.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42678.log.gz 2024-09-11T06:05:12.214 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25357.log 2024-09-11T06:05:12.214 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55827.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55827.log.gz 2024-09-11T06:05:12.215 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46263.log 2024-09-11T06:05:12.215 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.25357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25357.log.gz 2024-09-11T06:05:12.216 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40105.log 2024-09-11T06:05:12.217 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46263.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46263.log.gz 2024-09-11T06:05:12.217 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35524.log 2024-09-11T06:05:12.224 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40105.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40601.log.gz 2024-09-11T06:05:12.232 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36677.log 2024-09-11T06:05:12.237 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35524.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62004.log.gz 2024-09-11T06:05:12.238 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35524.log.gz 2024-09-11T06:05:12.239 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44586.log 2024-09-11T06:05:12.239 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29228.log 2024-09-11T06:05:12.240 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36677.log: /var/log/ceph/cluster2-client.mirror.44586.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36677.log.gz 2024-09-11T06:05:12.241 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50405.log 2024-09-11T06:05:12.241 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44586.log.gz 2024-09-11T06:05:12.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.29228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29228.log.gz 2024-09-11T06:05:12.242 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54661.log 2024-09-11T06:05:12.242 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51070.log 2024-09-11T06:05:12.243 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50405.log: /var/log/ceph/cluster2-client.mirror.54661.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54661.log.gz 2024-09-11T06:05:12.243 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50405.log.gz 2024-09-11T06:05:12.243 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32531.log 2024-09-11T06:05:12.243 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59130.log 2024-09-11T06:05:12.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51070.log: /var/log/ceph/cluster1-client.mirror.32531.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51070.log.gz 2024-09-11T06:05:12.244 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32531.log.gz 2024-09-11T06:05:12.245 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44307.log 2024-09-11T06:05:12.245 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36897.log 2024-09-11T06:05:12.246 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59130.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59130.log.gz 2024-09-11T06:05:12.246 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33914.log 2024-09-11T06:05:12.246 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.36897.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44307.log.gz 2024-09-11T06:05:12.246 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36897.log.gz 2024-09-11T06:05:12.247 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43584.log 2024-09-11T06:05:12.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61260.log 2024-09-11T06:05:12.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43584.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33914.log.gz 2024-09-11T06:05:12.249 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60819.log 2024-09-11T06:05:12.249 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43584.log.gz 2024-09-11T06:05:12.249 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61260.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61260.log.gz 2024-09-11T06:05:12.249 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45882.log 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28249.log 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60819.log.gz 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43449.log 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28249.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28249.log.gz 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45882.log.gz 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59040.log 2024-09-11T06:05:12.252 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52425.log 2024-09-11T06:05:12.253 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59040.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43449.log.gz 2024-09-11T06:05:12.253 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59040.log.gz 2024-09-11T06:05:12.253 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31055.log 2024-09-11T06:05:12.254 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34066.log 2024-09-11T06:05:12.254 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64519.log 2024-09-11T06:05:12.254 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52425.log.gz 2024-09-11T06:05:12.255 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55661.log 2024-09-11T06:05:12.255 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34066.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34066.log.gz 2024-09-11T06:05:12.255 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64519.log.gz 2024-09-11T06:05:12.255 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62810.log 2024-09-11T06:05:12.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35546.log 2024-09-11T06:05:12.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62810.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62810.log.gz 2024-09-11T06:05:12.257 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55661.log.gz 2024-09-11T06:05:12.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33153.log 2024-09-11T06:05:12.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57058.log 2024-09-11T06:05:12.259 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33153.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35546.log.gz 2024-09-11T06:05:12.259 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36611.log 2024-09-11T06:05:12.260 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33153.log.gz 2024-09-11T06:05:12.262 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57058.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57058.log.gz 2024-09-11T06:05:12.263 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31855.log 2024-09-11T06:05:12.263 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49967.log 2024-09-11T06:05:12.264 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31855.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31855.log.gz 2024-09-11T06:05:12.264 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55641.log 2024-09-11T06:05:12.265 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31055.log.gz 2024-09-11T06:05:12.265 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49967.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49967.log.gz 2024-09-11T06:05:12.266 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52951.log 2024-09-11T06:05:12.266 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56437.log 2024-09-11T06:05:12.267 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55641.log: /var/log/ceph/cluster1-client.mirror.52951.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55641.log.gz 2024-09-11T06:05:12.267 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52951.log.gz 2024-09-11T06:05:12.268 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59693.log 2024-09-11T06:05:12.268 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63569.log 2024-09-11T06:05:12.268 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56437.log: /var/log/ceph/cluster1-client.mirror.59693.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59693.log.gz 2024-09-11T06:05:12.269 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33955.log 2024-09-11T06:05:12.269 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63569.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63569.log.gz 2024-09-11T06:05:12.269 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36611.log.gz 2024-09-11T06:05:12.269 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44003.log 2024-09-11T06:05:12.269 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51287.log 2024-09-11T06:05:12.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44003.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56437.log.gz 2024-09-11T06:05:12.271 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56148.log 2024-09-11T06:05:12.271 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44003.log.gz 2024-09-11T06:05:12.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51287.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51287.log.gz 2024-09-11T06:05:12.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38147.log 2024-09-11T06:05:12.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51817.log 2024-09-11T06:05:12.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56148.log: /var/log/ceph/cluster2-client.mirror.38147.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56148.log.gz 2024-09-11T06:05:12.273 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36095.log 2024-09-11T06:05:12.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51817.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51817.log.gz 2024-09-11T06:05:12.274 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36396.log 2024-09-11T06:05:12.280 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36095.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36095.log.gz 2024-09-11T06:05:12.280 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59108.log 2024-09-11T06:05:12.286 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36396.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36396.log.gz 2024-09-11T06:05:12.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36877.log 2024-09-11T06:05:12.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59108.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59108.log.gz 2024-09-11T06:05:12.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62684.log 2024-09-11T06:05:12.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36877.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36877.log.gz 2024-09-11T06:05:12.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55537.log 2024-09-11T06:05:12.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62684.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62684.log.gz 2024-09-11T06:05:12.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31762.log 2024-09-11T06:05:12.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55537.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33955.log.gz 2024-09-11T06:05:12.293 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55537.log.gz 2024-09-11T06:05:12.293 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55227.log 2024-09-11T06:05:12.294 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.31762.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.31762.log.gz 2024-09-11T06:05:12.294 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64048.log 2024-09-11T06:05:12.298 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55227.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55227.log.gz 2024-09-11T06:05:12.298 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52183.log 2024-09-11T06:05:12.299 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64048.log.gz 2024-09-11T06:05:12.299 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.60532.log 2024-09-11T06:05:12.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52183.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52183.log.gz 2024-09-11T06:05:12.301 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34525.log 2024-09-11T06:05:12.307 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.60532.log: 93.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41257.log 2024-09-11T06:05:12.308 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34525.log: -- replaced with /var/log/ceph/cluster2-client.mirror.3.60532.log.gz 2024-09-11T06:05:12.314 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34525.log.gz 2024-09-11T06:05:12.316 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40303.log 2024-09-11T06:05:12.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45008.log 2024-09-11T06:05:12.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40303.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40303.log.gz 2024-09-11T06:05:12.343 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41427.log.gz 2024-09-11T06:05:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53630.log 2024-09-11T06:05:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45008.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45008.log.gz 2024-09-11T06:05:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55579.log 2024-09-11T06:05:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53630.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53630.log.gz 2024-09-11T06:05:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45179.log 2024-09-11T06:05:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55579.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55579.log.gz 2024-09-11T06:05:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54156.log 2024-09-11T06:05:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45179.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45179.log.gz 2024-09-11T06:05:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28152.log 2024-09-11T06:05:12.351 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54156.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54156.log.gz 2024-09-11T06:05:12.352 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60623.log 2024-09-11T06:05:12.352 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28152.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28152.log.gz 2024-09-11T06:05:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41606.log 2024-09-11T06:05:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60623.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60623.log.gz 2024-09-11T06:05:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50940.log 2024-09-11T06:05:12.356 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41606.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41606.log.gz 2024-09-11T06:05:12.356 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29371.log 2024-09-11T06:05:12.357 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50940.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50940.log.gz 2024-09-11T06:05:12.358 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64337.log 2024-09-11T06:05:12.358 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.29371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33633.log 2024-09-11T06:05:12.359 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64337.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.29371.log.gz 2024-09-11T06:05:12.360 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64337.log.gz 2024-09-11T06:05:12.360 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41934.log 2024-09-11T06:05:12.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33633.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33633.log.gz 2024-09-11T06:05:12.362 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32105.log 2024-09-11T06:05:12.363 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41934.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41934.log.gz 2024-09-11T06:05:12.363 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33290.log 2024-09-11T06:05:12.376 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42919.log 2024-09-11T06:05:12.378 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33290.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33290.log.gz 2024-09-11T06:05:12.385 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39216.log.gz 2024-09-11T06:05:12.392 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45926.log 2024-09-11T06:05:12.393 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49991.log 2024-09-11T06:05:12.393 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42919.log: /var/log/ceph/cluster1-client.mirror.45926.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42919.log.gz 2024-09-11T06:05:12.394 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45926.log.gz 2024-09-11T06:05:12.394 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44285.log 2024-09-11T06:05:12.395 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49991.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49991.log.gz 2024-09-11T06:05:12.395 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28361.log 2024-09-11T06:05:12.397 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44285.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44285.log.gz 2024-09-11T06:05:12.397 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28177.log 2024-09-11T06:05:12.398 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28361.log.gz 2024-09-11T06:05:12.398 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41384.log 2024-09-11T06:05:12.399 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58388.log 2024-09-11T06:05:12.400 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41384.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.admin.28177.log.gz 2024-09-11T06:05:12.401 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41384.log.gz 2024-09-11T06:05:12.401 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35344.log 2024-09-11T06:05:12.402 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58388.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58388.log.gz 2024-09-11T06:05:12.402 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54764.log 2024-09-11T06:05:12.403 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35344.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35344.log.gz 2024-09-11T06:05:12.403 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52145.log 2024-09-11T06:05:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54764.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54764.log.gz 2024-09-11T06:05:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41801.log 2024-09-11T06:05:12.408 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52145.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52145.log.gz 2024-09-11T06:05:12.408 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43825.log 2024-09-11T06:05:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41801.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41801.log.gz 2024-09-11T06:05:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55351.log 2024-09-11T06:05:12.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43825.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43825.log.gz 2024-09-11T06:05:12.411 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45251.log 2024-09-11T06:05:12.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55351.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55351.log.gz 2024-09-11T06:05:12.414 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37373.log 2024-09-11T06:05:12.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45251.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45251.log.gz 2024-09-11T06:05:12.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58905.log 2024-09-11T06:05:12.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.37373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38345.log 2024-09-11T06:05:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58905.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58905.log.gz 2024-09-11T06:05:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34964.log 2024-09-11T06:05:12.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38345.log: 89.6% 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38345.log.gz 2024-09-11T06:05:12.433 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37373.log.gz 2024-09-11T06:05:12.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.55096.log 2024-09-11T06:05:12.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34964.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34964.log.gz 2024-09-11T06:05:12.434 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59243.log 2024-09-11T06:05:12.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.55096.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.55096.log.gz 2024-09-11T06:05:12.440 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64441.log 2024-09-11T06:05:12.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59243.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59243.log.gz 2024-09-11T06:05:12.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53192.log 2024-09-11T06:05:12.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.64441.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64441.log.gz 2024-09-11T06:05:12.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61319.log 2024-09-11T06:05:12.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53192.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53192.log.gz 2024-09-11T06:05:12.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38088.log 2024-09-11T06:05:12.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52806.log 2024-09-11T06:05:12.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38088.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38088.log.gz 2024-09-11T06:05:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56831.log 2024-09-11T06:05:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52806.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52806.log.gz 2024-09-11T06:05:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55289.log 2024-09-11T06:05:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56831.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56831.log.gz 2024-09-11T06:05:12.484 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35746.log 2024-09-11T06:05:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55289.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55289.log.gz 2024-09-11T06:05:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45420.log 2024-09-11T06:05:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35746.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50473.log 2024-09-11T06:05:12.505 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35746.log.gz 2024-09-11T06:05:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45420.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45420.log.gz 2024-09-11T06:05:12.512 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61177.log 2024-09-11T06:05:12.513 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50473.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50473.log.gz 2024-09-11T06:05:12.520 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.62488.log 2024-09-11T06:05:12.521 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61177.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61177.log.gz 2024-09-11T06:05:12.528 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37884.log 2024-09-11T06:05:12.536 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.62488.log: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35632.log 2024-09-11T06:05:12.537 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37884.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37884.log.gz 2024-09-11T06:05:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37248.log 2024-09-11T06:05:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35632.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35632.log.gz 2024-09-11T06:05:12.548 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.62488.log.gz 2024-09-11T06:05:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50126.log 2024-09-11T06:05:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37248.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37248.log.gz 2024-09-11T06:05:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49242.log 2024-09-11T06:05:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50126.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50126.log.gz 2024-09-11T06:05:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40164.log 2024-09-11T06:05:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49242.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49242.log.gz 2024-09-11T06:05:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61520.log 2024-09-11T06:05:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40164.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40164.log.gz 2024-09-11T06:05:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49923.log 2024-09-11T06:05:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61520.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61520.log.gz 2024-09-11T06:05:12.576 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39024.log 2024-09-11T06:05:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49923.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49923.log.gz 2024-09-11T06:05:12.585 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62334.log 2024-09-11T06:05:12.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39024.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39024.log.gz 2024-09-11T06:05:12.592 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63757.log 2024-09-11T06:05:12.593 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62334.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62334.log.gz 2024-09-11T06:05:12.600 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61982.log 2024-09-11T06:05:12.601 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63757.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63757.log.gz 2024-09-11T06:05:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46673.log 2024-09-11T06:05:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61982.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61982.log.gz 2024-09-11T06:05:12.616 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47381.log 2024-09-11T06:05:12.617 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46673.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46673.log.gz 2024-09-11T06:05:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44066.log 2024-09-11T06:05:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47381.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47381.log.gz 2024-09-11T06:05:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59535.log 2024-09-11T06:05:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44066.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44066.log.gz 2024-09-11T06:05:12.641 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41702.log 2024-09-11T06:05:12.641 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59535.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59535.log.gz 2024-09-11T06:05:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51460.log 2024-09-11T06:05:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41702.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41702.log.gz 2024-09-11T06:05:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36548.log 2024-09-11T06:05:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51460.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51460.log.gz 2024-09-11T06:05:12.665 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32922.log 2024-09-11T06:05:12.666 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36548.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41257.log.gz 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36548.log.gz 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51359.log 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32922.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32922.log.gz 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34868.log 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51359.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51359.log.gz 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53570.log 2024-09-11T06:05:12.674 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34868.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34868.log.gz 2024-09-11T06:05:12.675 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55074.log 2024-09-11T06:05:12.676 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53570.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53570.log.gz 2024-09-11T06:05:12.676 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57828.log 2024-09-11T06:05:12.679 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55074.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55074.log.gz 2024-09-11T06:05:12.679 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34844.log 2024-09-11T06:05:12.680 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57828.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57828.log.gz 2024-09-11T06:05:12.680 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57802.log 2024-09-11T06:05:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34844.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34844.log.gz 2024-09-11T06:05:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42156.log 2024-09-11T06:05:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57802.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57802.log.gz 2024-09-11T06:05:12.685 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39523.log 2024-09-11T06:05:12.686 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42156.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42156.log.gz 2024-09-11T06:05:12.686 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49773.log 2024-09-11T06:05:12.687 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39523.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39523.log.gz 2024-09-11T06:05:12.687 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.35189.log 2024-09-11T06:05:12.688 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49773.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49773.log.gz 2024-09-11T06:05:12.688 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51663.log 2024-09-11T06:05:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.35189.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60878.log.gz 2024-09-11T06:05:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36118.log 2024-09-11T06:05:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51663.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51663.log.gz 2024-09-11T06:05:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41526.log 2024-09-11T06:05:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36118.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36118.log.gz 2024-09-11T06:05:12.712 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33536.log 2024-09-11T06:05:12.712 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41526.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41526.log.gz 2024-09-11T06:05:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59445.log 2024-09-11T06:05:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33536.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33536.log.gz 2024-09-11T06:05:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57637.log 2024-09-11T06:05:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59445.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59445.log.gz 2024-09-11T06:05:12.716 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51400.log 2024-09-11T06:05:12.717 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57637.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57637.log.gz 2024-09-11T06:05:12.717 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39397.log 2024-09-11T06:05:12.719 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51400.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51400.log.gz 2024-09-11T06:05:12.720 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54723.log 2024-09-11T06:05:12.720 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.39397.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39397.log.gz 2024-09-11T06:05:12.721 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59828.log 2024-09-11T06:05:12.722 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54723.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54723.log.gz 2024-09-11T06:05:12.722 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53674.log 2024-09-11T06:05:12.723 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59828.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59828.log.gz 2024-09-11T06:05:12.724 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55744.log 2024-09-11T06:05:12.724 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53674.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53674.log.gz 2024-09-11T06:05:12.725 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45362.log 2024-09-11T06:05:12.726 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55744.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55744.log.gz 2024-09-11T06:05:12.726 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36052.log 2024-09-11T06:05:12.728 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45362.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45362.log.gz 2024-09-11T06:05:12.728 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40442.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32734.log 2024-09-11T06:05:12.728 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.729 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36052.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61319.log.gz 2024-09-11T06:05:12.729 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37943.log 2024-09-11T06:05:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.32734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51962.log 2024-09-11T06:05:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37943.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32734.log.gz 2024-09-11T06:05:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62701.log 2024-09-11T06:05:12.731 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37943.log.gz 2024-09-11T06:05:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51962.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51962.log.gz 2024-09-11T06:05:12.732 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36376.log 2024-09-11T06:05:12.732 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59603.log 2024-09-11T06:05:12.733 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36376.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62701.log.gz 2024-09-11T06:05:12.733 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58410.log 2024-09-11T06:05:12.734 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59603.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36052.log.gz 2024-09-11T06:05:12.734 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59603.log.gz 2024-09-11T06:05:12.734 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48877.log 2024-09-11T06:05:12.735 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60778.log 2024-09-11T06:05:12.735 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58410.log.gz 2024-09-11T06:05:12.735 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52227.log 2024-09-11T06:05:12.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60778.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60778.log.gz 2024-09-11T06:05:12.737 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57596.log 2024-09-11T06:05:12.737 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48877.log.gz 2024-09-11T06:05:12.738 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60736.log 2024-09-11T06:05:12.738 INFO:teuthology.orchestra.run.smithi164.stderr: 93.6%/var/log/ceph/cluster2-client.mirror.57596.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52227.log.gz 2024-09-11T06:05:12.738 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36376.log.gz 2024-09-11T06:05:12.738 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53168.log 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57596.log.gz 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55310.log 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7%/var/log/ceph/cluster2-client.mirror.53168.log: -- replaced with /var/log/ceph/cluster1-client.mirror.60736.log.gz 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35860.log 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53168.log.gz 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32987.log 2024-09-11T06:05:12.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55310.log: /var/log/ceph/cluster2-client.mirror.35860.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55310.log.gz 2024-09-11T06:05:12.741 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57286.log 2024-09-11T06:05:12.741 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54291.log 2024-09-11T06:05:12.742 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57286.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57286.log.gz 2024-09-11T06:05:12.742 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37351.log 2024-09-11T06:05:12.743 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54291.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54291.log.gz 2024-09-11T06:05:12.743 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62721.log 2024-09-11T06:05:12.744 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37351.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37351.log.gz 2024-09-11T06:05:12.745 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51335.log 2024-09-11T06:05:12.745 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.35860.log.gz 2024-09-11T06:05:12.746 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62721.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62721.log.gz 2024-09-11T06:05:12.746 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32987.log.gz 2024-09-11T06:05:12.746 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52869.log 2024-09-11T06:05:12.747 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44172.log 2024-09-11T06:05:12.747 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51335.log: /var/log/ceph/cluster2-client.mirror.52869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53650.log 2024-09-11T06:05:12.747 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51335.log.gz 2024-09-11T06:05:12.747 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44172.log: 87.7% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44172.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.52869.log.gz 2024-09-11T06:05:12.747 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.748 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57182.log 2024-09-11T06:05:12.748 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42024.log 2024-09-11T06:05:12.748 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57182.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53650.log.gz 2024-09-11T06:05:12.749 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57513.log 2024-09-11T06:05:12.750 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41114.log 2024-09-11T06:05:12.750 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42024.log.gz 2024-09-11T06:05:12.750 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32850.log 2024-09-11T06:05:12.750 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41114.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41114.log.gz 2024-09-11T06:05:12.750 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57513.log.gz 2024-09-11T06:05:12.751 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21607.log 2024-09-11T06:05:12.752 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57182.log.gz/var/log/ceph/cluster1-client.mirror.32850.log: 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38044.log 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.32850.log.gz 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.tmp-client.admin.21607.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21607.log.gz 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35989.log 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34388.log 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35989.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38044.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62120.log 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40400.log 2024-09-11T06:05:12.754 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8%/var/log/ceph/cluster1-client.mirror.62120.log: -- replaced with /var/log/ceph/cluster1-client.mirror.34388.log.gz 2024-09-11T06:05:12.754 INFO:teuthology.orchestra.run.smithi164.stderr: 85.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62120.log.gz 2024-09-11T06:05:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.48105.log 2024-09-11T06:05:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56561.log 2024-09-11T06:05:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.48105.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40400.log.gz 2024-09-11T06:05:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42557.log 2024-09-11T06:05:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56561.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35989.log.gz 2024-09-11T06:05:12.758 INFO:teuthology.orchestra.run.smithi164.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56561.log.gz 2024-09-11T06:05:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44895.log 2024-09-11T06:05:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53389.log 2024-09-11T06:05:12.761 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44895.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42557.log.gz 2024-09-11T06:05:12.761 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44895.log.gz 2024-09-11T06:05:12.761 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48919.log 2024-09-11T06:05:12.762 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33558.log 2024-09-11T06:05:12.762 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48919.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53389.log.gz 2024-09-11T06:05:12.762 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48919.log.gz 2024-09-11T06:05:12.762 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59715.log 2024-09-11T06:05:12.763 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61277.log 2024-09-11T06:05:12.763 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33558.log: /var/log/ceph/cluster1-client.mirror.59715.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59715.log.gz 2024-09-11T06:05:12.764 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44389.log 2024-09-11T06:05:12.764 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61277.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61277.log.gz 2024-09-11T06:05:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32967.log 2024-09-11T06:05:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44389.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44389.log.gz 2024-09-11T06:05:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59760.log 2024-09-11T06:05:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32967.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32967.log.gz 2024-09-11T06:05:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36957.log 2024-09-11T06:05:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59760.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59760.log.gz 2024-09-11T06:05:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50497.log 2024-09-11T06:05:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36957.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36957.log.gz 2024-09-11T06:05:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60030.log 2024-09-11T06:05:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50497.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50497.log.gz 2024-09-11T06:05:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40063.log 2024-09-11T06:05:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60030.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60030.log.gz 2024-09-11T06:05:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43783.log 2024-09-11T06:05:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40063.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40063.log.gz 2024-09-11T06:05:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42105.log 2024-09-11T06:05:12.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43783.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43783.log.gz 2024-09-11T06:05:12.785 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43381.log 2024-09-11T06:05:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42105.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42105.log.gz 2024-09-11T06:05:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32555.log 2024-09-11T06:05:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43381.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43381.log.gz 2024-09-11T06:05:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46326.log 2024-09-11T06:05:12.789 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.32555.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32555.log.gz 2024-09-11T06:05:12.789 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49379.log 2024-09-11T06:05:12.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46326.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.48105.log.gz 2024-09-11T06:05:12.791 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46326.log.gz 2024-09-11T06:05:12.791 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55372.log 2024-09-11T06:05:12.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61839.log 2024-09-11T06:05:12.793 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55372.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55372.log.gz 2024-09-11T06:05:12.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45400.log 2024-09-11T06:05:12.794 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49379.log.gz 2024-09-11T06:05:12.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61839.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61839.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61862.log 2024-09-11T06:05:12.794 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50039.log 2024-09-11T06:05:12.795 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61862.log: 86.7% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45400.log.gz 2024-09-11T06:05:12.795 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61862.log.gz 2024-09-11T06:05:12.795 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53457.log 2024-09-11T06:05:12.796 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31075.log 2024-09-11T06:05:12.796 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53457.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50039.log.gz 2024-09-11T06:05:12.798 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53457.log.gz 2024-09-11T06:05:12.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42991.log 2024-09-11T06:05:12.798 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32105.log.gz/var/log/ceph/cluster1-client.mirror.31075.log: 2024-09-11T06:05:12.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58028.log 2024-09-11T06:05:12.798 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41870.log 2024-09-11T06:05:12.799 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.58028.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42991.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59625.log 2024-09-11T06:05:12.799 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:12.799 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58028.log.gz 2024-09-11T06:05:12.799 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41870.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41870.log.gz 2024-09-11T06:05:12.800 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38024.log 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37143.log 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59625.log.gz 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56189.log 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.37143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62578.log 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37143.log.gz 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56189.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38024.log.gz 2024-09-11T06:05:12.801 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40682.log 2024-09-11T06:05:12.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62578.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62578.log.gz 2024-09-11T06:05:12.802 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45998.log 2024-09-11T06:05:12.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40682.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56189.log.gz 2024-09-11T06:05:12.802 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30773.log 2024-09-11T06:05:12.802 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43208.log 2024-09-11T06:05:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45998.log: /var/log/ceph/cluster1-client.admin.30773.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45998.log.gz 2024-09-11T06:05:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57678.log 2024-09-11T06:05:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56872.log 2024-09-11T06:05:12.804 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43208.log.gz 2024-09-11T06:05:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43521.log 2024-09-11T06:05:12.807 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56872.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57678.log.gz 2024-09-11T06:05:12.807 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60216.log 2024-09-11T06:05:12.808 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4%/var/log/ceph/cluster1-client.mirror.43521.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56872.log.gz 2024-09-11T06:05:12.809 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43521.log.gz 2024-09-11T06:05:12.809 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46525.log 2024-09-11T06:05:12.809 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60798.log 2024-09-11T06:05:12.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46525.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46525.log.gz 2024-09-11T06:05:12.811 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33514.log 2024-09-11T06:05:12.811 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45685.log 2024-09-11T06:05:12.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60798.log: /var/log/ceph/cluster1-client.mirror.33514.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60798.log.gz 2024-09-11T06:05:12.816 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33514.log.gz 2024-09-11T06:05:12.817 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33935.log 2024-09-11T06:05:12.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60757.log 2024-09-11T06:05:12.818 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45685.log.gz 2024-09-11T06:05:12.819 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33935.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40105.log.gz 2024-09-11T06:05:12.823 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33935.log.gz 2024-09-11T06:05:12.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52203.log 2024-09-11T06:05:12.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43642.log 2024-09-11T06:05:12.824 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60757.log: /var/log/ceph/cluster2-client.mirror.52203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56127.log 2024-09-11T06:05:12.825 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60757.log.gz 2024-09-11T06:05:12.825 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52203.log.gz 2024-09-11T06:05:12.828 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43642.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43642.log.gz 2024-09-11T06:05:12.828 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44702.log 2024-09-11T06:05:12.829 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47140.log 2024-09-11T06:05:12.829 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57915.log 2024-09-11T06:05:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47140.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44702.log.gz 2024-09-11T06:05:12.830 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47140.log.gz 2024-09-11T06:05:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40820.log 2024-09-11T06:05:12.831 INFO:teuthology.orchestra.run.smithi164.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56127.log.gz 2024-09-11T06:05:12.831 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.30773.log.gz 2024-09-11T06:05:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57915.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57915.log.gz 2024-09-11T06:05:12.834 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63441.log 2024-09-11T06:05:12.834 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46938.log 2024-09-11T06:05:12.834 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40820.log: /var/log/ceph/cluster2-client.mirror.63441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54180.log 2024-09-11T06:05:12.834 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46938.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46938.log.gz 2024-09-11T06:05:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37168.log 2024-09-11T06:05:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54180.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54180.log.gz 2024-09-11T06:05:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52606.log 2024-09-11T06:05:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37168.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37168.log.gz 2024-09-11T06:05:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59085.log 2024-09-11T06:05:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52606.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31075.log.gz 2024-09-11T06:05:12.859 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52606.log.gz 2024-09-11T06:05:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50757.log 2024-09-11T06:05:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59085.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59085.log.gz 2024-09-11T06:05:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62643.log 2024-09-11T06:05:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50757.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50757.log.gz 2024-09-11T06:05:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39666.log 2024-09-11T06:05:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62643.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62643.log.gz 2024-09-11T06:05:12.899 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38147.log.gz 2024-09-11T06:05:12.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47921.log 2024-09-11T06:05:12.902 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39666.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39666.log.gz 2024-09-11T06:05:12.902 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56914.log 2024-09-11T06:05:12.903 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47921.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47921.log.gz 2024-09-11T06:05:12.903 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39174.log 2024-09-11T06:05:12.905 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56914.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56914.log.gz 2024-09-11T06:05:12.905 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57038.log 2024-09-11T06:05:12.906 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39174.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39174.log.gz 2024-09-11T06:05:12.907 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49620.log 2024-09-11T06:05:12.907 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57038.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57038.log.gz 2024-09-11T06:05:12.908 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53023.log 2024-09-11T06:05:12.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49620.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49620.log.gz 2024-09-11T06:05:12.910 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31711.log 2024-09-11T06:05:12.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53023.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53023.log.gz 2024-09-11T06:05:12.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29388.log 2024-09-11T06:05:12.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53505.log 2024-09-11T06:05:12.931 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.29388.log: 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29388.log.gz 2024-09-11T06:05:12.952 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57141.log 2024-09-11T06:05:12.953 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53505.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53505.log.gz 2024-09-11T06:05:12.960 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42155.log 2024-09-11T06:05:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57141.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57141.log.gz 2024-09-11T06:05:12.969 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36737.log 2024-09-11T06:05:12.969 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42155.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42155.log.gz 2024-09-11T06:05:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46766.log 2024-09-11T06:05:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36737.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36737.log.gz 2024-09-11T06:05:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63592.log 2024-09-11T06:05:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46766.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46766.log.gz 2024-09-11T06:05:13.009 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57720.log 2024-09-11T06:05:13.009 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63592.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63592.log.gz 2024-09-11T06:05:13.020 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32707.log 2024-09-11T06:05:13.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57720.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57720.log.gz 2024-09-11T06:05:13.032 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33451.log 2024-09-11T06:05:13.033 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.32707.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.32707.log.gz 2024-09-11T06:05:13.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59850.log 2024-09-11T06:05:13.056 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33451.log: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33100.log 2024-09-11T06:05:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59850.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59850.log.gz 2024-09-11T06:05:13.064 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33451.log.gz 2024-09-11T06:05:13.064 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29422.log 2024-09-11T06:05:13.065 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.33100.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.33100.log.gz 2024-09-11T06:05:13.072 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36505.log 2024-09-11T06:05:13.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.29422.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29422.log.gz 2024-09-11T06:05:13.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45320.log 2024-09-11T06:05:13.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36505.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40682.log.gz 2024-09-11T06:05:13.090 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36505.log.gz 2024-09-11T06:05:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51862.log 2024-09-11T06:05:13.093 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45320.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45320.log.gz 2024-09-11T06:05:13.093 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58433.log 2024-09-11T06:05:13.095 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51862.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51862.log.gz 2024-09-11T06:05:13.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55393.log 2024-09-11T06:05:13.096 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58433.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58433.log.gz 2024-09-11T06:05:13.096 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49049.log 2024-09-11T06:05:13.097 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55393.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55393.log.gz 2024-09-11T06:05:13.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59220.log 2024-09-11T06:05:13.098 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49049.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49049.log.gz 2024-09-11T06:05:13.099 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32300.log 2024-09-11T06:05:13.099 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59220.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59220.log.gz 2024-09-11T06:05:13.099 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54228.log 2024-09-11T06:05:13.112 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45030.log 2024-09-11T06:05:13.113 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54228.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54228.log.gz 2024-09-11T06:05:13.122 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33558.log.gz 2024-09-11T06:05:13.128 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56790.log 2024-09-11T06:05:13.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46167.log 2024-09-11T06:05:13.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56790.log: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45030.log.gz 2024-09-11T06:05:13.130 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56790.log.gz 2024-09-11T06:05:13.130 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42316.log 2024-09-11T06:05:13.131 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46167.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46167.log.gz 2024-09-11T06:05:13.131 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49439.log 2024-09-11T06:05:13.132 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42316.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42316.log.gz 2024-09-11T06:05:13.132 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57244.log 2024-09-11T06:05:13.134 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49439.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49439.log.gz 2024-09-11T06:05:13.134 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47800.log 2024-09-11T06:05:13.137 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57244.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57244.log.gz 2024-09-11T06:05:13.137 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40341.log 2024-09-11T06:05:13.139 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47800.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47800.log.gz 2024-09-11T06:05:13.139 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62234.log 2024-09-11T06:05:13.140 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40341.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40341.log.gz 2024-09-11T06:05:13.140 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44789.log 2024-09-11T06:05:13.141 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62234.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62234.log.gz 2024-09-11T06:05:13.141 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53216.log 2024-09-11T06:05:13.143 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44789.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44789.log.gz 2024-09-11T06:05:13.143 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58455.log 2024-09-11T06:05:13.144 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53216.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53216.log.gz 2024-09-11T06:05:13.144 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.41279.log 2024-09-11T06:05:13.145 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58455.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58455.log.gz 2024-09-11T06:05:13.145 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50978.log 2024-09-11T06:05:13.146 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59648.log 2024-09-11T06:05:13.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.41279.log: /var/log/ceph/cluster2-client.mirror.50978.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50978.log.gz 2024-09-11T06:05:13.149 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.41279.log.gz 2024-09-11T06:05:13.150 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31468.log 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44196.log 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59648.log: /var/log/ceph/cluster1-client.mirror.31468.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59648.log.gz 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.31468.log.gz 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56021.log 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53088.log 2024-09-11T06:05:13.153 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44196.log: /var/log/ceph/cluster2-client.mirror.56021.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44196.log.gz 2024-09-11T06:05:13.154 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56021.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54349.log 2024-09-11T06:05:13.154 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:13.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53088.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53088.log.gz 2024-09-11T06:05:13.155 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31513.log 2024-09-11T06:05:13.156 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31215.log 2024-09-11T06:05:13.157 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54349.log: /var/log/ceph/cluster2-client.mirror.31513.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54349.log.gz 2024-09-11T06:05:13.157 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37037.log 2024-09-11T06:05:13.158 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31215.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31215.log.gz 2024-09-11T06:05:13.159 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38937.log 2024-09-11T06:05:13.160 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37037.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37037.log.gz 2024-09-11T06:05:13.160 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.41934.log 2024-09-11T06:05:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38937.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38937.log.gz 2024-09-11T06:05:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43401.log 2024-09-11T06:05:13.166 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.41934.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.41934.log.gz 2024-09-11T06:05:13.167 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36009.log 2024-09-11T06:05:13.167 INFO:teuthology.orchestra.run.smithi164.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32300.log.gz 2024-09-11T06:05:13.168 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43401.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43401.log.gz 2024-09-11T06:05:13.168 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50104.log 2024-09-11T06:05:13.169 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.36009.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31513.log.gz 2024-09-11T06:05:13.173 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36009.log.gz 2024-09-11T06:05:13.174 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32898.log 2024-09-11T06:05:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52565.log 2024-09-11T06:05:13.175 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50104.log: /var/log/ceph/cluster1-client.mirror.32898.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32898.log.gz 2024-09-11T06:05:13.176 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50104.log.gz 2024-09-11T06:05:13.177 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30848.log 2024-09-11T06:05:13.177 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45641.log 2024-09-11T06:05:13.178 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52565.log: /var/log/ceph/cluster2-client.mirror.30848.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52565.log.gz 2024-09-11T06:05:13.178 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63699.log 2024-09-11T06:05:13.179 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43280.log 2024-09-11T06:05:13.180 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45641.log: /var/log/ceph/cluster2-client.mirror.63699.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63699.log.gz 2024-09-11T06:05:13.180 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45641.log.gz 2024-09-11T06:05:13.180 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46890.log 2024-09-11T06:05:13.181 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39947.log 2024-09-11T06:05:13.181 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46890.log: /var/log/ceph/cluster1-client.mirror.43280.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46890.log.gz 2024-09-11T06:05:13.181 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43280.log.gz 2024-09-11T06:05:13.181 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52058.log 2024-09-11T06:05:13.181 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47463.log 2024-09-11T06:05:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39947.log: /var/log/ceph/cluster1-client.mirror.52058.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52058.log.gz 2024-09-11T06:05:13.183 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40820.log.gz 2024-09-11T06:05:13.184 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53264.log 2024-09-11T06:05:13.184 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35481.log 2024-09-11T06:05:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47463.log: /var/log/ceph/cluster1-client.mirror.53264.log: 87.4% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53264.log.gz 2024-09-11T06:05:13.186 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47463.log.gz 2024-09-11T06:05:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50325.log 2024-09-11T06:05:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57451.log 2024-09-11T06:05:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62743.log 2024-09-11T06:05:13.188 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57451.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30848.log.gz 2024-09-11T06:05:13.188 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57451.log.gz 2024-09-11T06:05:13.189 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50325.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50325.log.gz 2024-09-11T06:05:13.190 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35481.log.gz 2024-09-11T06:05:13.191 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34568.log 2024-09-11T06:05:13.191 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52734.log 2024-09-11T06:05:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.34568.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34568.log.gz 2024-09-11T06:05:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28981.log 2024-09-11T06:05:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52734.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52734.log.gz 2024-09-11T06:05:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42899.log 2024-09-11T06:05:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.28981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35880.log 2024-09-11T06:05:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42899.log: 74.4% -- replaced with /var/log/ceph/cluster1-client.mirror.28981.log.gz 2024-09-11T06:05:13.199 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42899.log.gz 2024-09-11T06:05:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38526.log 2024-09-11T06:05:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35880.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35880.log.gz 2024-09-11T06:05:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39905.log 2024-09-11T06:05:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38526.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38526.log.gz 2024-09-11T06:05:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61945.log 2024-09-11T06:05:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39905.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39905.log.gz 2024-09-11T06:05:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48615.log 2024-09-11T06:05:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61945.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61945.log.gz 2024-09-11T06:05:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36917.log 2024-09-11T06:05:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48615.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48615.log.gz 2024-09-11T06:05:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61112.log 2024-09-11T06:05:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.36917.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36917.log.gz 2024-09-11T06:05:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37483.log 2024-09-11T06:05:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61112.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61112.log.gz 2024-09-11T06:05:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55434.log 2024-09-11T06:05:13.229 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47969.log 2024-09-11T06:05:13.229 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55434.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55434.log.gz 2024-09-11T06:05:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49680.log 2024-09-11T06:05:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47969.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47969.log.gz 2024-09-11T06:05:13.254 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.35189.log.gz 2024-09-11T06:05:13.256 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51504.log 2024-09-11T06:05:13.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49680.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49680.log.gz 2024-09-11T06:05:13.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44124.log 2024-09-11T06:05:13.259 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51504.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51504.log.gz 2024-09-11T06:05:13.268 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34985.log 2024-09-11T06:05:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44124.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44124.log.gz 2024-09-11T06:05:13.270 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57554.log 2024-09-11T06:05:13.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34985.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34985.log.gz 2024-09-11T06:05:13.276 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34688.log 2024-09-11T06:05:13.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57554.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57554.log.gz 2024-09-11T06:05:13.279 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53288.log 2024-09-11T06:05:13.280 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.34688.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34688.log.gz 2024-09-11T06:05:13.281 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30822.log 2024-09-11T06:05:13.282 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53288.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53288.log.gz 2024-09-11T06:05:13.282 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31711.log.gz 2024-09-11T06:05:13.282 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47053.log 2024-09-11T06:05:13.282 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.30822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61904.log 2024-09-11T06:05:13.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55959.log 2024-09-11T06:05:13.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47053.log.gz 2024-09-11T06:05:13.284 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61904.log.gz 2024-09-11T06:05:13.284 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59378.log 2024-09-11T06:05:13.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55959.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.30822.log.gz 2024-09-11T06:05:13.285 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29252.log 2024-09-11T06:05:13.285 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59378.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55959.log.gz 2024-09-11T06:05:13.285 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59378.log.gz 2024-09-11T06:05:13.285 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48897.log 2024-09-11T06:05:13.285 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.29252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33580.log 2024-09-11T06:05:13.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54930.log 2024-09-11T06:05:13.288 INFO:teuthology.orchestra.run.smithi164.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.29252.log.gz 2024-09-11T06:05:13.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.33580.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33580.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59490.log 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54930.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48897.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54930.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49748.log 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35114.log 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59490.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59490.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49748.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63634.log 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.49748.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.35114.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35114.log.gz 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48437.log 2024-09-11T06:05:13.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33175.log 2024-09-11T06:05:13.292 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63634.log.gz 2024-09-11T06:05:13.293 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37747.log 2024-09-11T06:05:13.293 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33175.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48437.log.gz 2024-09-11T06:05:13.293 INFO:teuthology.orchestra.run.smithi164.stderr: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33175.log.gz 2024-09-11T06:05:13.293 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32661.log 2024-09-11T06:05:13.293 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54826.log 2024-09-11T06:05:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32661.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32661.log.gz 2024-09-11T06:05:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35028.log 2024-09-11T06:05:13.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54826.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54826.log.gz 2024-09-11T06:05:13.297 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31533.log 2024-09-11T06:05:13.303 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35028.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35028.log.gz 2024-09-11T06:05:13.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39503.log 2024-09-11T06:05:13.309 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31533.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31533.log.gz 2024-09-11T06:05:13.309 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48636.log 2024-09-11T06:05:13.311 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39503.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39503.log.gz 2024-09-11T06:05:13.311 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46284.log 2024-09-11T06:05:13.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48636.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48636.log.gz 2024-09-11T06:05:13.314 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33406.log 2024-09-11T06:05:13.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46284.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46284.log.gz 2024-09-11T06:05:13.316 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33074.log 2024-09-11T06:05:13.317 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33406.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33406.log.gz 2024-09-11T06:05:13.318 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32619.log 2024-09-11T06:05:13.324 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48475.log 2024-09-11T06:05:13.342 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32619.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32619.log.gz 2024-09-11T06:05:13.344 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58725.log 2024-09-11T06:05:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48475.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48475.log.gz 2024-09-11T06:05:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37864.log 2024-09-11T06:05:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58725.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58725.log.gz 2024-09-11T06:05:13.361 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60216.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33382.log 2024-09-11T06:05:13.361 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:05:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37864.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37864.log.gz 2024-09-11T06:05:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46866.log 2024-09-11T06:05:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.33382.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33382.log.gz 2024-09-11T06:05:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49700.log 2024-09-11T06:05:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46866.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46866.log.gz 2024-09-11T06:05:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64249.log 2024-09-11T06:05:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49700.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49700.log.gz 2024-09-11T06:05:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56458.log 2024-09-11T06:05:13.381 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.64249.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64249.log.gz 2024-09-11T06:05:13.381 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50232.log 2024-09-11T06:05:13.383 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56458.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56458.log.gz 2024-09-11T06:05:13.384 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50781.log 2024-09-11T06:05:13.384 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50232.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50232.log.gz 2024-09-11T06:05:13.384 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39353.log 2024-09-11T06:05:13.385 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50781.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50781.log.gz 2024-09-11T06:05:13.385 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-11T06:05:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39353.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39353.log.gz 2024-09-11T06:05:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56520.log 2024-09-11T06:05:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37307.log 2024-09-11T06:05:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56520.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56520.log.gz 2024-09-11T06:05:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54204.log 2024-09-11T06:05:13.397 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61924.log 2024-09-11T06:05:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54204.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54204.log.gz 2024-09-11T06:05:13.399 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62422.log 2024-09-11T06:05:13.401 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61924.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61924.log.gz 2024-09-11T06:05:13.402 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61883.log 2024-09-11T06:05:13.403 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62422.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62422.log.gz 2024-09-11T06:05:13.403 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50805.log 2024-09-11T06:05:13.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61883.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61883.log.gz 2024-09-11T06:05:13.412 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57120.log 2024-09-11T06:05:13.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50805.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50805.log.gz 2024-09-11T06:05:13.414 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41685.log 2024-09-11T06:05:13.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57120.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62743.log.gz 2024-09-11T06:05:13.417 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57120.log.gz 2024-09-11T06:05:13.417 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51181.log 2024-09-11T06:05:13.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41685.log: 80.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40383.log 2024-09-11T06:05:13.418 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41685.log.gz 2024-09-11T06:05:13.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42819.log 2024-09-11T06:05:13.419 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51181.log.gz 2024-09-11T06:05:13.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42819.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42819.log.gz 2024-09-11T06:05:13.422 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40479.log 2024-09-11T06:05:13.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45661.log 2024-09-11T06:05:13.422 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40383.log.gz 2024-09-11T06:05:13.423 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56934.log 2024-09-11T06:05:13.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40479.log: /var/log/ceph/cluster2-client.mirror.45661.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45661.log.gz 2024-09-11T06:05:13.423 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57761.log 2024-09-11T06:05:13.424 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40479.log.gz 2024-09-11T06:05:13.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32041.log 2024-09-11T06:05:13.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57761.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57761.log.gz 2024-09-11T06:05:13.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60174.log 2024-09-11T06:05:13.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.32041.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56934.log.gz 2024-09-11T06:05:13.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60174.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60174.log.gz 2024-09-11T06:05:13.431 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49724.log 2024-09-11T06:05:13.431 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30795.log 2024-09-11T06:05:13.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49724.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49724.log.gz 2024-09-11T06:05:13.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41665.log 2024-09-11T06:05:13.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.30795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40704.log 2024-09-11T06:05:13.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41665.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.30795.log.gz 2024-09-11T06:05:13.438 INFO:teuthology.orchestra.run.smithi164.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32041.log.gz 2024-09-11T06:05:13.438 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49138.log 2024-09-11T06:05:13.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40704.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40704.log.gz 2024-09-11T06:05:13.439 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40933.log 2024-09-11T06:05:13.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49138.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49138.log.gz 2024-09-11T06:05:13.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47359.log 2024-09-11T06:05:13.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40933.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63441.log.gz 2024-09-11T06:05:13.458 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61385.log 2024-09-11T06:05:13.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47359.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47359.log.gz 2024-09-11T06:05:13.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58343.log 2024-09-11T06:05:13.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61385.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61385.log.gz 2024-09-11T06:05:13.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42509.log 2024-09-11T06:05:13.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58343.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58343.log.gz 2024-09-11T06:05:13.465 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55620.log 2024-09-11T06:05:13.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42509.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42509.log.gz 2024-09-11T06:05:13.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51311.log 2024-09-11T06:05:13.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55620.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55620.log.gz 2024-09-11T06:05:13.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50280.log 2024-09-11T06:05:13.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51311.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51311.log.gz 2024-09-11T06:05:13.469 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38306.log 2024-09-11T06:05:13.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.50280.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50280.log.gz 2024-09-11T06:05:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55496.log 2024-09-11T06:05:13.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54421.log 2024-09-11T06:05:13.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55496.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55496.log.gz 2024-09-11T06:05:13.504 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29610.log 2024-09-11T06:05:13.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54421.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54421.log.gz 2024-09-11T06:05:13.508 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46022.log 2024-09-11T06:05:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29610.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29610.log.gz 2024-09-11T06:05:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53891.log 2024-09-11T06:05:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46022.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46022.log.gz 2024-09-11T06:05:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58275.log 2024-09-11T06:05:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53891.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53891.log.gz 2024-09-11T06:05:13.537 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39947.log.gz 2024-09-11T06:05:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47580.log 2024-09-11T06:05:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58275.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58275.log.gz 2024-09-11T06:05:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39606.log 2024-09-11T06:05:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47580.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47580.log.gz 2024-09-11T06:05:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42461.log 2024-09-11T06:05:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39606.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39606.log.gz 2024-09-11T06:05:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54950.log 2024-09-11T06:05:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42461.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42461.log.gz 2024-09-11T06:05:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52517.log 2024-09-11T06:05:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54950.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54950.log.gz 2024-09-11T06:05:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56852.log 2024-09-11T06:05:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52517.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52517.log.gz 2024-09-11T06:05:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47339.log 2024-09-11T06:05:13.548 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37483.log.gz 2024-09-11T06:05:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56852.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56852.log.gz 2024-09-11T06:05:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44526.log 2024-09-11T06:05:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60532.log 2024-09-11T06:05:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47339.log: /var/log/ceph/cluster1-client.mirror.44526.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47339.log.gz 2024-09-11T06:05:13.552 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44526.log.gz 2024-09-11T06:05:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63929.log 2024-09-11T06:05:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60532.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60532.log.gz 2024-09-11T06:05:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59355.log 2024-09-11T06:05:13.555 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63929.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63929.log.gz 2024-09-11T06:05:13.555 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62622.log 2024-09-11T06:05:13.555 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59355.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59355.log.gz 2024-09-11T06:05:13.556 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54640.log 2024-09-11T06:05:13.556 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44104.log 2024-09-11T06:05:13.558 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62622.log: /var/log/ceph/cluster2-client.mirror.54640.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62622.log.gz 2024-09-11T06:05:13.558 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49822.log 2024-09-11T06:05:13.559 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44104.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44104.log.gz 2024-09-11T06:05:13.559 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54640.log.gz 2024-09-11T06:05:13.559 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37429.log 2024-09-11T06:05:13.560 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49863.log 2024-09-11T06:05:13.561 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49822.log: /var/log/ceph/cluster2-client.mirror.37429.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37429.log.gz 2024-09-11T06:05:13.561 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49822.log.gz 2024-09-11T06:05:13.561 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44220.log 2024-09-11T06:05:13.561 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44678.log 2024-09-11T06:05:13.562 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49863.log: /var/log/ceph/cluster1-client.mirror.44220.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44220.log.gz 2024-09-11T06:05:13.562 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49863.log.gz 2024-09-11T06:05:13.563 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40363.log 2024-09-11T06:05:13.563 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43883.log 2024-09-11T06:05:13.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44678.log: /var/log/ceph/cluster2-client.mirror.40363.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44678.log.gz 2024-09-11T06:05:13.564 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59310.log 2024-09-11T06:05:13.565 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43883.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43883.log.gz 2024-09-11T06:05:13.565 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28814.log 2024-09-11T06:05:13.566 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59310.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59310.log.gz 2024-09-11T06:05:13.566 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31010.log 2024-09-11T06:05:13.566 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.28814.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28814.log.gz 2024-09-11T06:05:13.567 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63007.log 2024-09-11T06:05:13.568 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49025.log 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63007.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63007.log.gz 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31010.log.gz 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41235.log 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49025.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49025.log.gz 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37805.log 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41235.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41235.log.gz 2024-09-11T06:05:13.573 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47487.log 2024-09-11T06:05:13.574 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.37805.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37805.log.gz 2024-09-11T06:05:13.574 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52445.log 2024-09-11T06:05:13.575 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47487.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47487.log.gz 2024-09-11T06:05:13.575 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35461.log 2024-09-11T06:05:13.576 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52445.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52445.log.gz 2024-09-11T06:05:13.576 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60122.log 2024-09-11T06:05:13.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35461.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35461.log.gz 2024-09-11T06:05:13.583 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-11T06:05:13.583 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60122.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60122.log.gz 2024-09-11T06:05:13.584 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57017.log 2024-09-11T06:05:13.584 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35189.log 2024-09-11T06:05:13.586 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57017.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57017.log.gz 2024-09-11T06:05:13.586 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47841.log 2024-09-11T06:05:13.597 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.35189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63115.log 2024-09-11T06:05:13.600 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47841.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47841.log.gz 2024-09-11T06:05:13.620 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38204.log 2024-09-11T06:05:13.621 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63115.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63115.log.gz 2024-09-11T06:05:13.622 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35416.log 2024-09-11T06:05:13.623 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.38204.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38204.log.gz 2024-09-11T06:05:13.636 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58500.log 2024-09-11T06:05:13.637 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35416.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35416.log.gz 2024-09-11T06:05:13.638 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52010.log 2024-09-11T06:05:13.638 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58500.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58500.log.gz 2024-09-11T06:05:13.652 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33774.log 2024-09-11T06:05:13.653 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52010.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52010.log.gz 2024-09-11T06:05:13.661 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52251.log 2024-09-11T06:05:13.661 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.33774.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33774.log.gz 2024-09-11T06:05:13.665 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41665.log.gz 2024-09-11T06:05:13.665 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29576.log 2024-09-11T06:05:13.666 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52251.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52251.log.gz 2024-09-11T06:05:13.667 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39316.log 2024-09-11T06:05:13.667 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.29576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45822.log 2024-09-11T06:05:13.668 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39316.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39316.log.gz 2024-09-11T06:05:13.668 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43425.log 2024-09-11T06:05:13.670 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45822.log: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29576.log.gz 2024-09-11T06:05:13.671 INFO:teuthology.orchestra.run.smithi164.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45822.log.gz 2024-09-11T06:05:13.671 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52299.log 2024-09-11T06:05:13.671 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43425.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43425.log.gz 2024-09-11T06:05:13.672 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41563.log 2024-09-11T06:05:13.672 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.52299.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52299.log.gz 2024-09-11T06:05:13.673 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31835.log 2024-09-11T06:05:13.673 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41563.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41563.log.gz 2024-09-11T06:05:13.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57430.log 2024-09-11T06:05:13.675 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.31835.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31835.log.gz 2024-09-11T06:05:13.675 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56893.log 2024-09-11T06:05:13.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57430.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57430.log.gz 2024-09-11T06:05:13.678 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31951.log 2024-09-11T06:05:13.679 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56893.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56893.log.gz 2024-09-11T06:05:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62276.log 2024-09-11T06:05:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.31951.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31951.log.gz 2024-09-11T06:05:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47704.log 2024-09-11T06:05:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62276.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62276.log.gz 2024-09-11T06:05:13.682 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39745.log 2024-09-11T06:05:13.682 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47704.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47704.log.gz 2024-09-11T06:05:13.683 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43714.log 2024-09-11T06:05:13.684 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39745.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39745.log.gz 2024-09-11T06:05:13.685 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42726.log 2024-09-11T06:05:13.685 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43714.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43714.log.gz 2024-09-11T06:05:13.686 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35440.log 2024-09-11T06:05:13.686 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42726.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42726.log.gz 2024-09-11T06:05:13.688 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.35440.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35440.log.gz 2024-09-11T06:05:13.812 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35189.log.gz 2024-09-11T06:05:13.863 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38306.log.gz 2024-09-11T06:05:13.897 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40933.log.gz 2024-09-11T06:05:13.904 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37747.log.gz 2024-09-11T06:05:13.979 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40363.log.gz 2024-09-11T06:05:14.037 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33074.log.gz 2024-09-11T06:05:14.048 INFO:teuthology.orchestra.run.smithi164.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37307.log.gz 2024-09-11T06:05:17.827 INFO:teuthology.orchestra.run.smithi067.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-11T06:05:18.299 INFO:teuthology.orchestra.run.smithi067.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-11T06:05:18.508 INFO:teuthology.orchestra.run.smithi067.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-11T06:05:18.509 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-11T06:05:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m9.686s 2024-09-11T06:05:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m28.845s 2024-09-11T06:05:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m1.008s 2024-09-11T06:06:12.553 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-11T06:06:18.670 INFO:teuthology.orchestra.run.smithi164.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-11T06:06:37.054 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-11T06:06:37.057 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-11T06:06:37.057 INFO:teuthology.orchestra.run.smithi164.stderr:real 1m28.212s 2024-09-11T06:06:37.057 INFO:teuthology.orchestra.run.smithi164.stderr:user 3m46.146s 2024-09-11T06:06:37.057 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m8.998s 2024-09-11T06:06:37.058 INFO:tasks.ceph:Archiving logs... 2024-09-11T06:06:37.059 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi067/log 2024-09-11T06:06:37.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T06:06:40.430 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi164/log 2024-09-11T06:06:40.430 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T06:07:07.602 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T06:07:07.617 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T06:07:07.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T06:07:07.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T06:07:07.682 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T06:07:07.697 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T06:07:07.697 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:07:07.700 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:07:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:07:07.710 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-09-11T06:07:07.710 INFO:teuthology.orchestra.run.smithi067.stdout:*hv01.front.sepi 67.205.162.81 3 u 60 64 377 0.088 -2.911 0.169 2024-09-11T06:07:07.710 INFO:teuthology.orchestra.run.smithi067.stdout:+hv02.front.sepi 63.231.80.2 3 u 61 64 377 0.075 -2.158 0.143 2024-09-11T06:07:07.710 INFO:teuthology.orchestra.run.smithi067.stdout:+hv03.front.sepi 74.6.168.72 3 u 56 64 377 0.107 -2.814 0.303 2024-09-11T06:07:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout:+hv01.front.sepi 67.205.162.81 3 u 48 64 377 0.094 -1.588 0.138 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout:+hv02.front.sepi 63.231.80.2 3 u 61 64 377 0.111 -1.195 0.167 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout:*hv03.front.sepi 74.6.168.72 3 u 60 64 377 0.097 -1.412 0.105 2024-09-11T06:07:07.741 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T06:07:07.742 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T06:07:07.754 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T06:07:07.754 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T06:07:07.765 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T06:07:07.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T06:07:07.788 INFO:teuthology.task.internal:Duration was 3140.164685 seconds 2024-09-11T06:07:07.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T06:07:07.800 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T06:07:07.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T06:07:07.802 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T06:07:07.965 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T06:07:07.965 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-09-11T06:07:07.965 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T06:07:07.976 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-09-11T06:07:07.976 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T06:07:07.987 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T06:07:07.987 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T06:07:08.021 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T06:07:08.060 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T06:07:08.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T06:07:08.097 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T06:07:08.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T06:07:08.264 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T06:07:08.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T06:07:08.278 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T06:07:08.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T06:07:08.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:07:08.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:07:08.351 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-09-11T06:07:08.368 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-09-11T06:07:08.398 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:07:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:07:08.428 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:07:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:07:08.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T06:07:08.458 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T06:07:08.459 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi067 2024-09-11T06:07:08.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T06:07:08.491 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900545/remote/smithi164 2024-09-11T06:07:08.492 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T06:07:08.513 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T06:07:08.514 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T06:07:08.533 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T06:07:08.561 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T06:07:08.574 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T06:07:08.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T06:07:08.608 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T06:07:08.608 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T06:07:08.610 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T06:07:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 06:07 /home/ubuntu/cephtest 2024-09-11T06:07:08.616 INFO:teuthology.orchestra.run.smithi164.stdout: 658464 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 06:07 /home/ubuntu/cephtest 2024-09-11T06:07:08.617 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T06:07:08.630 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T06:07:08.660 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3140.16468501091 failure_reason: '"2024-09-11T05:27:35.354983+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-11T06:07:08.660 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T06:07:08.745 INFO:teuthology.run:FAIL